2014-10-28 23:44:28 +01:00
|
|
|
|
Quick start
|
|
|
|
|
-----------
|
|
|
|
|
|
|
|
|
|
Requirements
|
|
|
|
|
............
|
|
|
|
|
|
2017-06-02 22:59:35 +02:00
|
|
|
|
- Django 1.8, 1.10 or 1.11
|
|
|
|
|
- Python 2.7, 3.4, 3.5 or 3.6
|
2014-12-14 10:12:38 +01:00
|
|
|
|
- a cache configured as ``'default'`` with one of these backends:
|
2014-12-08 03:47:11 +01:00
|
|
|
|
|
2017-06-04 18:37:39 +02:00
|
|
|
|
- `django-redis <https://github.com/niwinz/django-redis>`_
|
|
|
|
|
- `memcached <https://docs.djangoproject.com/en/1.11/topics/cache/#memcached>`_
|
2015-04-11 01:57:13 +02:00
|
|
|
|
(using either python-memcached or pylibmc)
|
2017-06-04 18:37:39 +02:00
|
|
|
|
- `filebased <https://docs.djangoproject.com/en/1.11/topics/cache/#filesystem-caching>`_
|
|
|
|
|
- `locmem <https://docs.djangoproject.com/en/1.11/topics/cache/#local-memory-caching>`_
|
2015-10-25 20:08:18 +01:00
|
|
|
|
(but it’s not shared between processes, see :ref:`locmem limits <Locmem>`)
|
2014-12-08 03:47:11 +01:00
|
|
|
|
|
2014-12-13 20:05:39 +01:00
|
|
|
|
- one of these databases:
|
|
|
|
|
|
|
|
|
|
- PostgreSQL
|
|
|
|
|
- SQLite
|
2016-09-29 19:12:01 +02:00
|
|
|
|
- MySQL (but on older versions like MySQL 5.5, django-cachalot has no effect,
|
2015-10-25 20:08:18 +01:00
|
|
|
|
see :ref:`MySQL limits <MySQL>`)
|
2014-10-28 23:44:28 +01:00
|
|
|
|
|
|
|
|
|
Usage
|
|
|
|
|
.....
|
|
|
|
|
|
|
|
|
|
#. ``pip install django-cachalot``
|
|
|
|
|
#. Add ``'cachalot',`` to your ``INSTALLED_APPS``
|
2015-10-25 00:47:24 +02:00
|
|
|
|
#. If you use multiple servers with a common cache server,
|
|
|
|
|
:ref:`double check their clock synchronisation <multiple servers>`
|
2015-10-25 20:08:18 +01:00
|
|
|
|
#. If you modify data outside Django
|
2016-09-29 19:12:01 +02:00
|
|
|
|
– typically after restoring a SQL database –,
|
|
|
|
|
use the :ref:`manage.py command <Command>`
|
2016-09-13 13:51:22 +02:00
|
|
|
|
#. Be aware of :ref:`the few other limits <Limits>`
|
2014-12-08 22:00:08 +01:00
|
|
|
|
#. If you use
|
2016-09-29 19:12:01 +02:00
|
|
|
|
`django-debug-toolbar <https://github.com/jazzband/django-debug-toolbar>`_,
|
2014-12-13 20:05:39 +01:00
|
|
|
|
you can add ``'cachalot.panels.CachalotPanel',``
|
|
|
|
|
to your ``DEBUG_TOOLBAR_PANELS``
|
2014-10-28 23:44:28 +01:00
|
|
|
|
#. Enjoy!
|
|
|
|
|
|
|
|
|
|
|
2015-10-25 20:08:18 +01:00
|
|
|
|
.. _Settings:
|
|
|
|
|
|
2014-10-28 23:44:28 +01:00
|
|
|
|
Settings
|
|
|
|
|
........
|
|
|
|
|
|
2014-10-30 04:17:48 +01:00
|
|
|
|
``CACHALOT_ENABLED``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``True``
|
|
|
|
|
:Description: If set to ``False``, disables SQL caching but keeps invalidating
|
2016-09-29 19:12:01 +02:00
|
|
|
|
to avoid stale cache.
|
2014-10-30 04:17:48 +01:00
|
|
|
|
|
|
|
|
|
``CACHALOT_CACHE``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``'default'``
|
2016-09-29 19:12:01 +02:00
|
|
|
|
:Description:
|
|
|
|
|
Alias of the cache from |CACHES|_ used by django-cachalot.
|
|
|
|
|
|
|
|
|
|
.. warning::
|
|
|
|
|
After modifying this setting, you should invalidate the cache
|
|
|
|
|
:ref:`using the manage.py command <Command>` or :ref:`the API <API>`.
|
|
|
|
|
Indeed, only the cache configured using this setting is automatically
|
|
|
|
|
invalidated by django-cachalot – for optimisation reasons. So when you
|
|
|
|
|
change this setting, you end up on a cache that may contain stale data.
|
2014-10-28 23:44:28 +01:00
|
|
|
|
|
|
|
|
|
.. |CACHES| replace:: ``CACHES``
|
2017-06-04 18:37:39 +02:00
|
|
|
|
.. _CACHES: https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-CACHES
|
|
|
|
|
|
|
|
|
|
``CACHALOT_DATABASES``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``'supported_only'``
|
|
|
|
|
:Description:
|
|
|
|
|
List, tuple, set or frozenset of database aliases from |DATABASES|_ against
|
|
|
|
|
which django-cachalot will do caching. By default, the special value
|
|
|
|
|
``'supported_only'`` enables django-cachalot only on supported database
|
|
|
|
|
engines.
|
|
|
|
|
|
|
|
|
|
.. |DATABASES| replace:: ``DATABASES``
|
|
|
|
|
.. _DATABASES: https://docs.djangoproject.com/en/1.11/ref/settings/#std:setting-DATABASES
|
2014-10-28 23:44:28 +01:00
|
|
|
|
|
2016-09-06 21:57:26 +02:00
|
|
|
|
``CACHALOT_TIMEOUT``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``None``
|
|
|
|
|
:Description:
|
|
|
|
|
Number of seconds during which the cache should consider data as valid.
|
|
|
|
|
``None`` means an infinite timeout.
|
|
|
|
|
|
|
|
|
|
.. warning::
|
|
|
|
|
Cache timeouts don’t work in a strict way on most cache backends.
|
2016-09-07 00:57:27 +02:00
|
|
|
|
A cache might not keep data during the requested timeout:
|
2016-09-06 21:57:26 +02:00
|
|
|
|
it can keep it in memory during a shorter time than the specified timeout.
|
|
|
|
|
It can even keep it longer, even if data is not returned when you request it.
|
|
|
|
|
So **don’t rely on timeouts to limit the size of your database**,
|
|
|
|
|
you might face some unexpected behaviour.
|
|
|
|
|
Always set the maximum cache size instead.
|
|
|
|
|
|
2014-10-30 04:17:48 +01:00
|
|
|
|
``CACHALOT_CACHE_RANDOM``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``False``
|
|
|
|
|
:Description: If set to ``True``, caches random queries
|
2016-09-29 19:12:01 +02:00
|
|
|
|
(those with ``order_by('?')``).
|
2014-10-30 04:17:48 +01:00
|
|
|
|
|
2014-11-04 01:17:35 +01:00
|
|
|
|
.. _CACHALOT_INVALIDATE_RAW:
|
|
|
|
|
|
|
|
|
|
``CACHALOT_INVALIDATE_RAW``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``True``
|
2016-09-29 19:12:01 +02:00
|
|
|
|
:Description:
|
|
|
|
|
If set to ``False``, disables automatic invalidation on raw
|
|
|
|
|
SQL queries – read :ref:`raw queries limits <Raw SQL queries>` for more info.
|
2014-11-04 01:17:35 +01:00
|
|
|
|
|
2015-10-05 20:09:10 +02:00
|
|
|
|
|
|
|
|
|
``CACHALOT_ONLY_CACHABLE_TABLES``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``frozenset()``
|
|
|
|
|
:Description:
|
|
|
|
|
Sequence of SQL table names that will be the only ones django-cachalot
|
|
|
|
|
will cache. Only queries with a subset of these tables will be cached.
|
2015-10-25 00:47:24 +02:00
|
|
|
|
The sequence being empty (as it is by default) doesn’t mean that no table
|
2016-09-06 21:57:26 +02:00
|
|
|
|
can be cached: it disables this setting, so any table can be cached.
|
2015-10-25 00:47:24 +02:00
|
|
|
|
:ref:`CACHALOT_UNCACHABLE_TABLES` has more weight than this:
|
|
|
|
|
if you add a table to both settings, it will never be cached.
|
2016-09-07 00:57:27 +02:00
|
|
|
|
Run ``./manage.py invalidate_cachalot`` after changing this setting.
|
2015-10-05 20:09:10 +02:00
|
|
|
|
|
|
|
|
|
|
2015-10-04 18:35:44 +02:00
|
|
|
|
.. _CACHALOT_UNCACHABLE_TABLES:
|
|
|
|
|
|
2015-06-17 21:24:49 +02:00
|
|
|
|
``CACHALOT_UNCACHABLE_TABLES``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``frozenset(('django_migrations',))``
|
|
|
|
|
:Description:
|
|
|
|
|
Sequence of SQL table names that will be ignored by django-cachalot.
|
|
|
|
|
Queries using a table mentioned in this setting will not be cached.
|
|
|
|
|
Always keep ``'django_migrations'`` in it, otherwise you may face
|
|
|
|
|
some issues, especially during tests.
|
2016-09-07 00:57:27 +02:00
|
|
|
|
Run ``./manage.py invalidate_cachalot`` after changing this setting.
|
2015-06-17 21:24:49 +02:00
|
|
|
|
|
2014-10-30 04:17:48 +01:00
|
|
|
|
``CACHALOT_QUERY_KEYGEN``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``'cachalot.utils.get_query_cache_key'``
|
|
|
|
|
:Description: Python module path to the function that will be used to generate
|
2016-09-07 00:57:27 +02:00
|
|
|
|
the cache key of a SQL query.
|
|
|
|
|
Run ``./manage.py invalidate_cachalot``
|
|
|
|
|
after changing this setting.
|
2014-10-30 04:17:48 +01:00
|
|
|
|
|
|
|
|
|
``CACHALOT_TABLE_KEYGEN``
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
:Default: ``'cachalot.utils.get_table_cache_key'``
|
|
|
|
|
:Description: Python module path to the function that will be used to generate
|
2016-09-07 00:57:27 +02:00
|
|
|
|
the cache key of a SQL table.
|
|
|
|
|
Clear your cache after changing this setting (it’s not enough
|
|
|
|
|
to use ``./manage.py invalidate_cachalot``).
|
2014-10-30 04:17:48 +01:00
|
|
|
|
|
2016-09-29 19:12:01 +02:00
|
|
|
|
|
|
|
|
|
.. _Command:
|
|
|
|
|
|
|
|
|
|
``manage.py`` command
|
|
|
|
|
.....................
|
|
|
|
|
|
|
|
|
|
``manage.py invalidate_cachalot`` is available to invalidate all the cache keys
|
|
|
|
|
set by django-cachalot. If you run it without any argument, it invalidates all
|
|
|
|
|
models on all caches and all databases. But you can specify what applications
|
|
|
|
|
or models are invalidated, and on which cache or database.
|
|
|
|
|
|
|
|
|
|
Examples:
|
|
|
|
|
|
|
|
|
|
``./manage.py invalidate_cachalot auth``
|
|
|
|
|
Invalidates all models from the 'auth' application.
|
|
|
|
|
``./manage.py invalidate_cachalot your_app auth.User``
|
|
|
|
|
Invalidates all models from the 'your_app' application, but also
|
|
|
|
|
the ``User`` model from the 'auth' application.
|
|
|
|
|
``./manage.py invalidate_cachalot -c redis -p postgresql``
|
|
|
|
|
Invalidates all models,
|
|
|
|
|
but only for the database configured with the 'postgresql' alias,
|
|
|
|
|
and only for the cache configured with the 'redis' alias.
|
|
|
|
|
|
|
|
|
|
|
2016-09-13 20:15:30 +02:00
|
|
|
|
.. _Template utils:
|
2015-10-25 20:08:18 +01:00
|
|
|
|
|
2016-09-13 20:15:30 +02:00
|
|
|
|
Template utils
|
|
|
|
|
..............
|
2015-10-25 00:47:24 +02:00
|
|
|
|
|
|
|
|
|
`Caching template fragments <https://docs.djangoproject.com/en/1.8/topics/cache/#template-fragment-caching>`_
|
|
|
|
|
can be extremely powerful to speedup a Django application. However, it often
|
|
|
|
|
means you have to adapt your models to get a relevant cache key, typically
|
|
|
|
|
by adding a timestamp that refers to the last modification of the object.
|
|
|
|
|
|
|
|
|
|
But modifying your models and caching template fragments leads
|
|
|
|
|
to stale contents most of the time. There’s a simple reason to that: we rarely
|
|
|
|
|
only display the data from one model, we often want to display related data,
|
|
|
|
|
such as the number of books written by someone, display a quote from a book
|
|
|
|
|
of this author, display similar authors, etc. In such situations,
|
|
|
|
|
**it’s impossible to cache template fragments and avoid stale rendered data**.
|
|
|
|
|
|
|
|
|
|
Fortunately, django-cachalot provides an easy way to fix this issue,
|
|
|
|
|
by simply checking when was the last time data changed in the given models
|
|
|
|
|
or tables. The API function
|
|
|
|
|
:meth:`get_last_invalidation <cachalot.api.get_last_invalidation>` does that,
|
|
|
|
|
and we provided a ``get_last_invalidation`` template tag to directly
|
|
|
|
|
use it in templates. It works exactly the same as the API function.
|
|
|
|
|
|
2016-09-13 20:15:30 +02:00
|
|
|
|
Django template tag
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
2015-10-25 00:47:24 +02:00
|
|
|
|
Example of a quite heavy nested loop with a lot of SQL queries
|
|
|
|
|
(considering no prefetch has been done)::
|
|
|
|
|
|
|
|
|
|
{% load cachalot cache %}
|
|
|
|
|
|
|
|
|
|
{% get_last_invalidation 'auth.User' 'library.Book' 'library.Author' as last_invalidation %}
|
|
|
|
|
{% cache 3600 short_user_profile last_invalidation %}
|
|
|
|
|
{{ user }} has borrowed these books:
|
|
|
|
|
{% for book in user.borrowed_books.all %}
|
|
|
|
|
<div class="book">
|
|
|
|
|
{{ book }} ({{ book.pages.count }} pages)
|
|
|
|
|
<span class="authors">
|
|
|
|
|
{% for author in book.authors.all %}
|
|
|
|
|
{{ author }}{% if not forloop.last %},{% endif %}
|
|
|
|
|
{% endfor %}
|
|
|
|
|
</span>
|
|
|
|
|
</div>
|
|
|
|
|
{% endfor %}
|
|
|
|
|
{% endcache %}
|
|
|
|
|
|
|
|
|
|
``cache_alias`` and ``db_alias`` keywords arguments of this template tag
|
|
|
|
|
are also available (see
|
|
|
|
|
:meth:`cachalot.api.get_last_invalidation`).
|
|
|
|
|
|
2016-09-13 20:15:30 +02:00
|
|
|
|
Jinja2 statement and function
|
|
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
|
|
|
|
|
|
A Jinja2 extension for django-cachalot can be used, simply add
|
2017-06-02 22:57:31 +02:00
|
|
|
|
``'cachalot.jinja2ext.cachalot',`` to the ``'extensions'`` list of the ``OPTIONS``
|
2016-09-13 20:15:30 +02:00
|
|
|
|
dict in the Django ``TEMPLATES`` settings.
|
|
|
|
|
|
|
|
|
|
It provides:
|
|
|
|
|
|
|
|
|
|
- The API function
|
|
|
|
|
:meth:`get_last_invalidation <cachalot.api.get_last_invalidation>` directly
|
|
|
|
|
available as a function anywhere in Jinja2.
|
|
|
|
|
- An Jinja2 statement equivalent to the ``cache`` template tag of Django.
|
|
|
|
|
|
|
|
|
|
The ``cache`` does the same thing as its Django template equivalent,
|
|
|
|
|
except that ``cache_key`` and ``timeout`` are optional keyword arguments, and
|
|
|
|
|
you need to add commas between arguments. When unspecified, ``cache_key`` is
|
|
|
|
|
generated from the template filename plus the statement line number, and
|
|
|
|
|
``timeout`` defaults to infinite. To specify which cache should store the
|
|
|
|
|
saved content, use the ``cache_alias`` keyword argument.
|
|
|
|
|
|
|
|
|
|
Same example than above, but for Jinja2::
|
|
|
|
|
|
|
|
|
|
{% cache get_last_invalidation('auth.User', 'library.Book', 'library.Author'),
|
|
|
|
|
cache_key='short_user_profile', timeout=3600 %}
|
|
|
|
|
{{ user }} has borrowed these books:
|
|
|
|
|
{% for book in user.borrowed_books.all() %}
|
|
|
|
|
<div class="book">
|
|
|
|
|
{{ book }} ({{ book.pages.count() }} pages)
|
|
|
|
|
<span class="authors">
|
|
|
|
|
{% for author in book.authors.all() %}
|
|
|
|
|
{{ author }}{% if not loop.last %},{% endif %}
|
|
|
|
|
{% endfor %}
|
|
|
|
|
</span>
|
|
|
|
|
</div>
|
|
|
|
|
{% endfor %}
|
|
|
|
|
{% endcache %}
|
|
|
|
|
|
|
|
|
|
|
2015-10-25 20:08:18 +01:00
|
|
|
|
.. _Signal:
|
|
|
|
|
|
2015-04-17 20:06:18 +02:00
|
|
|
|
Signal
|
|
|
|
|
......
|
|
|
|
|
|
|
|
|
|
``cachalot.signals.post_invalidation`` is available if you need to do something
|
|
|
|
|
just after a cache invalidation (when you modify something in a SQL table).
|
|
|
|
|
``sender`` is the name of the SQL table invalidated, and a keyword argument
|
|
|
|
|
``db_alias`` explains which database is affected by the invalidation.
|
|
|
|
|
Be careful when you specify ``sender``, as it is sensible to string type.
|
|
|
|
|
To be sure, use ``Model._meta.db_table``.
|
|
|
|
|
|
2015-10-28 12:54:39 +01:00
|
|
|
|
This signal is not directly triggered during transactions,
|
|
|
|
|
it waits until the current transaction ends. This signal is also triggered
|
|
|
|
|
when invalidating using the API or the ``manage.py`` command. Be careful
|
|
|
|
|
when using multiple databases, if you invalidate all databases by simply
|
|
|
|
|
calling ``invalidate()``, this signal will be triggered one time
|
|
|
|
|
for each database and for each model. If you have 3 databases and 20 models,
|
|
|
|
|
``invalidate()`` will trigger the signal 60 times.
|
|
|
|
|
|
2015-04-17 20:06:18 +02:00
|
|
|
|
Example:
|
|
|
|
|
|
|
|
|
|
.. code:: python
|
|
|
|
|
|
|
|
|
|
from cachalot.signals import post_invalidation
|
|
|
|
|
from django.dispatch import receiver
|
|
|
|
|
from django.core.mail import mail_admins
|
|
|
|
|
from django.contrib.auth import *
|
|
|
|
|
|
|
|
|
|
# This prints a message to the console after each table invalidation
|
|
|
|
|
def invalidation_debug(sender, **kwargs):
|
|
|
|
|
db_alias = kwargs['db_alias']
|
|
|
|
|
print('%s was invalidated in the DB configured as %s'
|
|
|
|
|
% (sender, db_alias))
|
|
|
|
|
|
|
|
|
|
post_invalidation.connect(invalidation_debug)
|
|
|
|
|
|
|
|
|
|
# Using the `receiver` decorator is just a nicer way
|
|
|
|
|
# to write the same thing as `signal.connect`.
|
|
|
|
|
# Here we specify `sender` so that the function is executed only if
|
|
|
|
|
# the table invalidated is the one specified.
|
|
|
|
|
# We also connect it several times to be executed for several senders.
|
|
|
|
|
@receiver(post_invalidation, sender=User.groups.through._meta.db_table)
|
|
|
|
|
@receiver(post_invalidation, sender=User.user_permissions.through._meta.db_table)
|
|
|
|
|
@receiver(post_invalidation, sender=Group.permissions.through._meta.db_table)
|
|
|
|
|
def warn_admin(sender, **kwargs):
|
|
|
|
|
mail_admins('User permissions changed',
|
|
|
|
|
'Someone probably gained or lost Django permissions.')
|