1095 lines
33 KiB
Plaintext
1095 lines
33 KiB
Plaintext
.. _changelog:
|
|
|
|
================
|
|
Change history
|
|
================
|
|
|
|
This document contains change notes for bugfix releases in the 3.1.x series
|
|
(Cipater), please see :ref:`whatsnew-3.1` for an overview of what's
|
|
new in Celery 3.1.
|
|
|
|
.. _version-3.1.13:
|
|
|
|
3.1.13
|
|
======
|
|
|
|
Security Fixes
|
|
--------------
|
|
|
|
* [Security: `CELERYSA-0002`_] Insecure default umask.
|
|
|
|
The built-in utility used to daemonize the Celery worker service sets
|
|
an insecure umask by default (umask 0).
|
|
|
|
This means that any files or directories created by the worker will
|
|
end up having world-writable permissions.
|
|
|
|
Special thanks to Red Hat for originally discovering and reporting the
|
|
issue!
|
|
|
|
This version will no longer set a default umask by default, so if unset
|
|
the umask of the parent process will be used.
|
|
|
|
.. _`CELERYSA-0002`:
|
|
http://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0002.txt
|
|
|
|
News
|
|
----
|
|
|
|
- **Requirements**
|
|
|
|
- Now depends on :ref:`Kombu 3.0.21 <kombu:version-3.0.21>`.
|
|
|
|
- Now depends on :mod:`billiard` 3.3.0.18.
|
|
|
|
|
|
- **App**: ``backend`` argument now also sets the :setting:`CELERY_RESULT_BACKEND`
|
|
setting.
|
|
|
|
- **Task**: ``signature_from_request`` now propagates ``reply_to`` so that
|
|
the RPC backend works with retried tasks (Issue #2113).
|
|
|
|
- **Task**: ``retry`` will no longer attempt to requeue the task if sending
|
|
the retry message fails.
|
|
|
|
Unrelated exceptions being raised could cause a message loop, so it was
|
|
better to remove this behavior.
|
|
|
|
- **Beat**: Accounts for standard 1ms drift by always waking up 0.010s
|
|
earlier.
|
|
|
|
This will adjust the latency so that the periodic tasks will not move
|
|
1ms after every invocation.
|
|
|
|
- Documentation fixes
|
|
|
|
Contributed by Yuval Greenfield, Lucas Wiman, nicholsonjf
|
|
|
|
- **Worker**: Removed an outdated assert statement that could lead to errors
|
|
being masked (Issue #2086).
|
|
|
|
|
|
|
|
.. _version-3.1.12:
|
|
|
|
3.1.12
|
|
======
|
|
:release-date: 2014-06-09 10:12 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- **Requirements**
|
|
|
|
Now depends on :ref:`Kombu 3.0.19 <kombu:version-3.0.19>`.
|
|
|
|
- **App**: Connections were not being closed after fork due to an error in the
|
|
after fork handler (Issue #2055).
|
|
|
|
This could manifest itself by causing framing errors when using RabbitMQ.
|
|
(``Unexpected frame``).
|
|
|
|
- **Django**: ``django.setup()`` was being called too late when
|
|
using Django 1.7 (Issue #1802).
|
|
|
|
- **Django**: Fixed problems with event timezones when using Django
|
|
(``Substantial drift``).
|
|
|
|
Celery did not take into account that Django modifies the
|
|
``time.timeone`` attributes and friends.
|
|
|
|
- **Canvas**: ``Signature.link`` now works when the link option is a scalar
|
|
value (Issue #2019).
|
|
|
|
- **Prefork pool**: Fixed race conditions for when file descriptors are
|
|
removed from the event loop.
|
|
|
|
Fix contributed by Roger Hu.
|
|
|
|
- **Prefork pool**: Improved solution for dividing tasks between child
|
|
processes.
|
|
|
|
This change should improve performance when there are many child
|
|
processes, and also decrease the chance that two subsequent tasks are
|
|
written to the same child process.
|
|
|
|
- **Worker**: Now ignores unknown event types, instead of crashing.
|
|
|
|
Fix contributed by Illes Solt.
|
|
|
|
- **Programs**: :program:`celery worker --detach` no longer closes open file
|
|
descriptors when :envvar:`C_FAKEFORK` is used so that the workers output
|
|
can be seen.
|
|
|
|
- **Programs**: The default working directory for :program:`celery worker
|
|
--detach` is now the current working directory, not ``/``.
|
|
|
|
- **Canvas**: ``signature(s, app=app)`` did not upgrade serialized signatures
|
|
to their original class (``subtask_type``) when the ``app`` keyword argument
|
|
was used.
|
|
|
|
- **Control**: The ``duplicate nodename`` warning emitted by control commands
|
|
now shows the duplicate node name.
|
|
|
|
- **Tasks**: Can now call ``ResultSet.get()`` on a result set without members.
|
|
|
|
Fix contributed by Alexey Kotlyarov.
|
|
|
|
- **App**: Fixed strange traceback mangling issue for
|
|
``app.connection_or_acquire``.
|
|
|
|
- **Programs**: The :program:`celery multi stopwait` command is now documented
|
|
in usage.
|
|
|
|
- **Other**: Fixed cleanup problem with ``PromiseProxy`` when an error is
|
|
raised while trying to evaluate the promise.
|
|
|
|
- **Other**: The utility used to censor configuration values now handles
|
|
non-string keys.
|
|
|
|
Fix contributed by Luke Pomfrey.
|
|
|
|
- **Other**: The ``inspect conf`` command did not handle non-string keys well.
|
|
|
|
Fix contributed by Jay Farrimond.
|
|
|
|
- **Programs**: Fixed argument handling problem in
|
|
:program:`celery worker --detach`.
|
|
|
|
Fix contributed by Dmitry Malinovsky.
|
|
|
|
- **Programs**: :program:`celery worker --detach` did not forward working
|
|
directory option (Issue #2003).
|
|
|
|
- **Programs**: :program:`celery inspect registered` no longer includes
|
|
the list of built-in tasks.
|
|
|
|
- **Worker**: The ``requires`` attribute for boot steps were not being handled
|
|
correctly (Issue #2002).
|
|
|
|
- **Eventlet**: The eventlet pool now supports the ``pool_grow`` and
|
|
``pool_shrink`` remote control commands.
|
|
|
|
Contributed by Mher Movsisyan.
|
|
|
|
- **Eventlet**: The eventlet pool now implements statistics for
|
|
:program:``celery inspect stats``.
|
|
|
|
Contributed by Mher Movsisyan.
|
|
|
|
- **Documentation**: Clarified ``Task.rate_limit`` behavior.
|
|
|
|
Contributed by Jonas Haag.
|
|
|
|
- **Documentation**: ``AbortableTask`` examples now updated to use the new
|
|
API (Issue #1993).
|
|
|
|
- **Documentation**: The security documentation examples used an out of date
|
|
import.
|
|
|
|
Fix contributed by Ian Dees.
|
|
|
|
- **Init scripts**: The CentOS init scripts did not quote
|
|
:envvar:`CELERY_CHDIR`.
|
|
|
|
Fix contributed by ffeast.
|
|
|
|
.. _version-3.1.11:
|
|
|
|
3.1.11
|
|
======
|
|
:release-date: 2014-04-16 11:00 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- **Now compatible with RabbitMQ 3.3.0**
|
|
|
|
You need to run Celery 3.1.11 or later when using RabbitMQ 3.3,
|
|
and if you use the ``librabbitmq`` module you also have to upgrade
|
|
to librabbitmq 1.5.0:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ pip install -U librabbitmq
|
|
|
|
- **Requirements**:
|
|
|
|
- Now depends on :ref:`Kombu 3.0.15 <kombu:version-3.0.15>`.
|
|
|
|
- Now depends on `billiard 3.3.0.17`_.
|
|
|
|
- Bundle ``celery[librabbitmq]`` now depends on :mod:`librabbitmq` 1.5.0.
|
|
|
|
.. _`billiard 3.3.0.17`:
|
|
https://github.com/celery/billiard/blob/master/CHANGES.txt
|
|
|
|
- **Tasks**: The :setting:`CELERY_DEFAULT_DELIVERY_MODE` setting was being
|
|
ignored (Issue #1953).
|
|
|
|
- **Worker**: New :option:`--heartbeat-interval` can be used to change the
|
|
time (in seconds) between sending event heartbeats.
|
|
|
|
Contributed by Matthew Duggan and Craig Northway.
|
|
|
|
- **App**: Fixed memory leaks occurring when creating lots of temporary
|
|
app instances (Issue #1949).
|
|
|
|
- **MongoDB**: SSL configuration with non-MongoDB transport breaks MongoDB
|
|
results backend (Issue #1973).
|
|
|
|
Fix contributed by Brian Bouterse.
|
|
|
|
- **Logging**: The color formatter accidentally modified ``record.msg``
|
|
(Issue #1939).
|
|
|
|
- **Results**: Fixed problem with task trails being stored multiple times,
|
|
causing ``result.collect()`` to hang (Issue #1936, Issue #1943).
|
|
|
|
- **Results**: ``ResultSet`` now implements a ``.backend`` attribute for
|
|
compatibility with ``AsyncResult``.
|
|
|
|
- **Results**: ``.forget()`` now also clears the local cache.
|
|
|
|
- **Results**: Fixed problem with multiple calls to ``result._set_cache``
|
|
(Issue #1940).
|
|
|
|
- **Results**: ``join_native`` populated result cache even if disabled.
|
|
|
|
- **Results**: The YAML result serializer should now be able to handle storing
|
|
exceptions.
|
|
|
|
- **Worker**: No longer sends task error emails for expected errors (in
|
|
``@task(throws=(..., )))``.
|
|
|
|
- **Canvas**: Fixed problem with exception deserialization when using
|
|
the JSON serializer (Issue #1987).
|
|
|
|
- **Eventlet**: Fixes crash when ``celery.contrib.batches`` attempted to
|
|
cancel a non-existing timer (Issue #1984).
|
|
|
|
- Can now import ``celery.version_info_t``, and ``celery.five`` (Issue #1968).
|
|
|
|
|
|
.. _version-3.1.10:
|
|
|
|
3.1.10
|
|
======
|
|
:release-date: 2014-03-22 09:40 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- **Requirements**:
|
|
|
|
- Now depends on :ref:`Kombu 3.0.14 <kombu:version-3.0.14>`.
|
|
|
|
- **Results**:
|
|
|
|
Reliability improvements to the SQLAlchemy database backend. Previously the
|
|
connection from the MainProcess was improperly shared with the workers.
|
|
(Issue #1786)
|
|
|
|
- **Redis:** Important note about events (Issue #1882).
|
|
|
|
There is a new transport option for Redis that enables monitors
|
|
to filter out unwanted events. Enabling this option in the workers
|
|
will increase performance considerably:
|
|
|
|
.. code-block:: python
|
|
|
|
BROKER_TRANSPORT_OPTIONS = {'fanout_patterns': True}
|
|
|
|
Enabling this option means that your workers will not be able to see
|
|
workers with the option disabled (or is running an older version of
|
|
Celery), so if you do enable it then make sure you do so on all
|
|
nodes.
|
|
|
|
See :ref:`redis-caveats-fanout-patterns`.
|
|
|
|
This will be the default in Celery 3.2.
|
|
|
|
- **Results**: The :class:`@AsyncResult` object now keeps a local cache
|
|
of the final state of the task.
|
|
|
|
This means that the global result cache can finally be disabled,
|
|
and you can do so by setting :setting:`CELERY_MAX_CACHED_RESULTS` to
|
|
:const:`-1`. The lifetime of the cache will then be bound to the
|
|
lifetime of the result object, which will be the default behavior
|
|
in Celery 3.2.
|
|
|
|
- **Events**: The "Substantial drift" warning message is now logged once
|
|
per node name only (Issue #1802).
|
|
|
|
- **Worker**: Ability to use one log file per child process when using the
|
|
prefork pool.
|
|
|
|
This can be enabled by using the new ``%i`` and ``%I`` format specifiers
|
|
for the log file name. See :ref:`worker-files-process-index`.
|
|
|
|
- **Redis**: New experimental chord join implementation.
|
|
|
|
This is an optimization for chords when using the Redis result backend,
|
|
where the join operation is now considerably faster and using less
|
|
resources than the previous strategy.
|
|
|
|
The new option can be set in the result backend URL:
|
|
|
|
CELERY_RESULT_BACKEND = 'redis://localhost?new_join=1'
|
|
|
|
This must be enabled manually as it's incompatible
|
|
with workers and clients not using it, so be sure to enable
|
|
the option in all clients and workers if you decide to use it.
|
|
|
|
- **Multi**: With ``-opt:index`` (e.g. :option:`-c:1`) the index now always refers
|
|
to the position of a node in the argument list.
|
|
|
|
This means that referring to a number will work when specifying a list
|
|
of node names and not just for a number range:
|
|
|
|
.. code-block:: bash
|
|
|
|
celery multi start A B C D -c:1 4 -c:2-4 8
|
|
|
|
In this example ``1`` refers to node A (as it's the first node in the
|
|
list).
|
|
|
|
- **Signals**: The sender argument to ``Signal.connect`` can now be a proxy
|
|
object, which means that it can be used with the task decorator
|
|
(Issue #1873).
|
|
|
|
- **Task**: A regression caused the ``queue`` argument to ``Task.retry`` to be
|
|
ignored (Issue #1892).
|
|
|
|
- **App**: Fixed error message for :meth:`~@Celery.config_from_envvar`.
|
|
|
|
Fix contributed by Dmitry Malinovsky.
|
|
|
|
- **Canvas**: Chords can now contain a group of other chords (Issue #1921).
|
|
|
|
- **Canvas**: Chords can now be combined when using the amqp result backend
|
|
(a chord where the callback is also a chord).
|
|
|
|
- **Canvas**: Calling ``result.get()`` for a chain task will now complete
|
|
even if one of the tasks in the chain is ``ignore_result=True``
|
|
(Issue #1905).
|
|
|
|
- **Canvas**: Worker now also logs chord errors.
|
|
|
|
- **Canvas**: A chord task raising an exception will now result in
|
|
any errbacks (``link_error``) to the chord callback to also be called.
|
|
|
|
- **Results**: Reliability improvements to the SQLAlchemy database backend
|
|
(Issue #1786).
|
|
|
|
Previously the connection from the ``MainProcess`` was improperly
|
|
inherited by child processes.
|
|
|
|
Fix contributed by Ionel Cristian Mărieș.
|
|
|
|
- **Task**: Task callbacks and errbacks are now called using the group
|
|
primitive.
|
|
|
|
- **Task**: ``Task.apply`` now properly sets ``request.headers``
|
|
(Issue #1874).
|
|
|
|
- **Worker**: Fixed ``UnicodeEncodeError`` occuring when worker is started
|
|
by `supervisord`.
|
|
|
|
Fix contributed by Codeb Fan.
|
|
|
|
- **Beat**: No longer attempts to upgrade a newly created database file
|
|
(Issue #1923).
|
|
|
|
- **Beat**: New setting :setting:``CELERYBEAT_SYNC_EVERY`` can be be used
|
|
to control file sync by specifying the number of tasks to send between
|
|
each sync.
|
|
|
|
Contributed by Chris Clark.
|
|
|
|
- **Commands**: :program:`celery inspect memdump` no longer crashes
|
|
if the :mod:`psutil` module is not installed (Issue #1914).
|
|
|
|
- **Worker**: Remote control commands now always accepts json serialized
|
|
messages (Issue #1870).
|
|
|
|
- **Worker**: Gossip will now drop any task related events it receives
|
|
by mistake (Issue #1882).
|
|
|
|
|
|
.. _version-3.1.9:
|
|
|
|
3.1.9
|
|
=====
|
|
:release-date: 2014-02-10 06:43 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- **Requirements**:
|
|
|
|
- Now depends on :ref:`Kombu 3.0.12 <kombu:version-3.0.12>`.
|
|
|
|
- **Prefork pool**: Better handling of exiting child processes.
|
|
|
|
Fix contributed by Ionel Cristian Mărieș.
|
|
|
|
- **Prefork pool**: Now makes sure all file descriptors are removed
|
|
from the hub when a process is cleaned up.
|
|
|
|
Fix contributed by Ionel Cristian Mărieș.
|
|
|
|
- **New Sphinx extension**: for autodoc documentation of tasks:
|
|
:mod:`celery.contrib.spinx` (Issue #1833).
|
|
|
|
- **Django**: Now works with Django 1.7a1.
|
|
|
|
- **Task**: Task.backend is now a property that forwards to ``app.backend``
|
|
if no custom backend has been specified for the task (Issue #1821).
|
|
|
|
- **Generic init scripts**: Fixed bug in stop command.
|
|
|
|
Fix contributed by Rinat Shigapov.
|
|
|
|
- **Generic init scripts**: Fixed compatibility with GNU :manpage:`stat`.
|
|
|
|
Fix contributed by Paul Kilgo.
|
|
|
|
- **Generic init scripts**: Fixed compatibility with the minimal
|
|
:program:`dash` shell (Issue #1815).
|
|
|
|
- **Commands**: The :program:`celery amqp basic.publish` command was not
|
|
working properly.
|
|
|
|
Fix contributed by Andrey Voronov.
|
|
|
|
- **Commands**: Did no longer emit an error message if the pidfile exists
|
|
and the process is still alive (Issue #1855).
|
|
|
|
- **Commands**: Better error message for missing arguments to preload
|
|
options (Issue #1860).
|
|
|
|
- **Commands**: :program:`celery -h` did not work because of a bug in the
|
|
argument parser (Issue #1849).
|
|
|
|
- **Worker**: Improved error message for message decoding errors.
|
|
|
|
- **Time**: Now properly parses the `Z` timezone specifier in ISO 8601 date
|
|
strings.
|
|
|
|
Fix contributed by Martin Davidsson.
|
|
|
|
- **Worker**: Now uses the *negotiated* heartbeat value to calculate
|
|
how often to run the heartbeat checks.
|
|
|
|
- **Beat**: Fixed problem with beat hanging after the first schedule
|
|
iteration (Issue #1822).
|
|
|
|
Fix contributed by Roger Hu.
|
|
|
|
- **Signals**: The header argument to :signal:`before_task_publish` is now
|
|
always a dictionary instance so that signal handlers can add headers.
|
|
|
|
- **Worker**: A list of message headers is now included in message related
|
|
errors.
|
|
|
|
.. _version-3.1.8:
|
|
|
|
3.1.8
|
|
=====
|
|
:release-date: 2014-01-17 10:45 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- **Requirements**:
|
|
|
|
- Now depends on :ref:`Kombu 3.0.10 <kombu:version-3.0.10>`.
|
|
|
|
- Now depends on `billiard 3.3.0.14`_.
|
|
|
|
.. _`billiard 3.3.0.14`:
|
|
https://github.com/celery/billiard/blob/master/CHANGES.txt
|
|
|
|
- **Worker**: The event loop was not properly reinitialized at consumer restart
|
|
which would force the worker to continue with a closed ``epoll`` instance on
|
|
Linux, resulting in a crash.
|
|
|
|
- **Events:** Fixed issue with both heartbeats and task events that could
|
|
result in the data not being kept in sorted order.
|
|
|
|
As a result this would force the worker to log "heartbeat missed"
|
|
events even though the remote node was sending heartbeats in a timely manner.
|
|
|
|
- **Results:** The pickle serializer no longer converts group results to tuples,
|
|
and will keep the original type (*Issue #1750*).
|
|
|
|
- **Results:** ``ResultSet.iterate`` is now pending deprecation.
|
|
|
|
The method will be deprecated in version 3.2 and removed in version 3.3.
|
|
|
|
Use ``result.get(callback=)`` (or ``result.iter_native()`` where available)
|
|
instead.
|
|
|
|
- **Worker**\|eventlet/gevent: A regression caused ``Ctrl+C`` to be ineffective
|
|
for shutdown.
|
|
|
|
- **Redis result backend:** Now using a pipeline to store state changes
|
|
for improved performance.
|
|
|
|
Contributed by Pepijn de Vos.
|
|
|
|
- **Redis result backend:** Will now retry storing the result if disconnected.
|
|
|
|
- **Worker**\|gossip: Fixed attribute error occurring when another node leaves.
|
|
|
|
Fix contributed by Brodie Rao.
|
|
|
|
- **Generic init scripts:** Now runs a check at startup to verify
|
|
that any configuration scripts are owned by root and that they
|
|
are not world/group writeable.
|
|
|
|
The init script configuration is a shell script executed by root,
|
|
so this is a preventive measure to ensure that users do not
|
|
leave this file vulnerable to changes by unprivileged users.
|
|
|
|
.. note::
|
|
|
|
Note that upgrading celery will not update the init scripts,
|
|
instead you need to manually copy the improved versions from the
|
|
source distribution:
|
|
https://github.com/celery/celery/tree/3.1/extra/generic-init.d
|
|
|
|
- **Commands**: The :program:`celery purge` command now warns that the operation
|
|
will delete all tasks and prompts the user for confirmation.
|
|
|
|
A new :option:`-f` was added that can be used to disable
|
|
interactive mode.
|
|
|
|
- **Task**: ``.retry()`` did not raise the value provided in the ``exc`` argument
|
|
when called outside of an error context (*Issue #1755*).
|
|
|
|
- **Commands:** The :program:`celery multi` command did not forward command
|
|
line configuration to the target workers.
|
|
|
|
The change means that multi will forward the special ``--`` argument and
|
|
configuration content at the end of the arguments line to the specified
|
|
workers.
|
|
|
|
Example using command-line configuration to set a broker heartbeat
|
|
from :program:`celery multi`:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ celery multi start 1 -c3 -- broker.heartbeat=30
|
|
|
|
Fix contributed by Antoine Legrand.
|
|
|
|
- **Canvas:** ``chain.apply_async()`` now properly forwards execution options.
|
|
|
|
Fix contributed by Konstantin Podshumok.
|
|
|
|
- **Redis result backend:** Now takes ``connection_pool`` argument that can be
|
|
used to change the connection pool class/constructor.
|
|
|
|
- **Worker:** Now truncates very long arguments and keyword arguments logged by
|
|
the pool at debug severity.
|
|
|
|
- **Worker:** The worker now closes all open files on :sig:`SIGHUP` (regression)
|
|
(*Issue #1768*).
|
|
|
|
Fix contributed by Brodie Rao
|
|
|
|
- **Worker:** Will no longer accept remote control commands while the
|
|
worker startup phase is incomplete (*Issue #1741*).
|
|
|
|
- **Commands:** The output of the event dump utility
|
|
(:program:`celery events -d`) can now be piped into other commands.
|
|
|
|
- **Documentation:** The RabbitMQ installation instructions for OS X was
|
|
updated to use modern homebrew practices.
|
|
|
|
Contributed by Jon Chen.
|
|
|
|
- **Commands:** The :program:`celery inspect conf` utility now works.
|
|
|
|
- **Commands:** The :option:`-no-color` argument was not respected by
|
|
all commands (*Issue #1799*).
|
|
|
|
- **App:** Fixed rare bug with ``autodiscover_tasks()`` (*Issue #1797*).
|
|
|
|
- **Distribution:** The sphinx docs will now always add the parent directory
|
|
to path so that the current celery source code is used as a basis for
|
|
API documentation (*Issue #1782*).
|
|
|
|
- **Documentation:** Supervisord examples contained an extraneous '-' in a
|
|
`--logfile` argument example.
|
|
|
|
Fix contributed by Mohammad Almeer.
|
|
|
|
.. _version-3.1.7:
|
|
|
|
3.1.7
|
|
=====
|
|
:release-date: 2013-12-17 06:00 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
.. _v317-important:
|
|
|
|
Important Notes
|
|
---------------
|
|
|
|
Init script security improvements
|
|
---------------------------------
|
|
|
|
Where the generic init scripts (for ``celeryd``, and ``celerybeat``) before
|
|
delegated the responsibility of dropping privileges to the target application,
|
|
it will now use ``su`` instead, so that the Python program is not trusted
|
|
with superuser privileges.
|
|
|
|
This is not in reaction to any known exploit, but it will
|
|
limit the possibility of a privilege escalation bug being abused in the
|
|
future.
|
|
|
|
You have to upgrade the init scripts manually from this directory:
|
|
https://github.com/celery/celery/tree/3.1/extra/generic-init.d
|
|
|
|
AMQP result backend
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
The 3.1 release accidentally left the amqp backend configured to be
|
|
non-persistent by default.
|
|
|
|
Upgrading from 3.0 would give a "not equivalent" error when attempting to
|
|
set or retrieve results for a task. That is unless you manually set the
|
|
persistence setting::
|
|
|
|
CELERY_RESULT_PERSISTENT = True
|
|
|
|
This version restores the previous value so if you already forced
|
|
the upgrade by removing the existing exchange you must either
|
|
keep the configuration by setting ``CELERY_RESULT_PERSISTENT = False``
|
|
or delete the ``celeryresults`` exchange again.
|
|
|
|
Synchronous subtasks
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Tasks waiting for the result of a subtask will now emit
|
|
a :exc:`RuntimeWarning` warning when using the prefork pool,
|
|
and in 3.2 this will result in an exception being raised.
|
|
|
|
It's not legal for tasks to block by waiting for subtasks
|
|
as this is likely to lead to resource starvation and eventually
|
|
deadlock when using the prefork pool (see also :ref:`task-synchronous-subtasks`).
|
|
|
|
If you really know what you are doing you can avoid the warning (and
|
|
the future exception being raised) by moving the operation in a whitelist
|
|
block:
|
|
|
|
.. code-block:: python
|
|
|
|
from celery.result import allow_join_result
|
|
|
|
@app.task
|
|
def misbehaving():
|
|
result = other_task.delay()
|
|
with allow_join_result():
|
|
result.get()
|
|
|
|
Note also that if you wait for the result of a subtask in any form
|
|
when using the prefork pool you must also disable the pool prefetching
|
|
behavior with the worker :ref:`-Ofair option <prefork-pool-prefetch>`.
|
|
|
|
.. _v317-fixes:
|
|
|
|
Fixes
|
|
-----
|
|
|
|
- Now depends on :ref:`Kombu 3.0.8 <kombu:version-3.0.8>`.
|
|
|
|
- Now depends on :mod:`billiard` 3.3.0.13
|
|
|
|
- Events: Fixed compatibility with non-standard json libraries
|
|
that sends float as :class:`decimal.Decimal` (Issue #1731)
|
|
|
|
- Events: State worker objects now always defines attributes:
|
|
``active``, ``processed``, ``loadavg``, ``sw_ident``, ``sw_ver``
|
|
and ``sw_sys``.
|
|
|
|
- Worker: Now keeps count of the total number of tasks processed,
|
|
not just by type (``all_active_count``).
|
|
|
|
- Init scripts: Fixed problem with reading configuration file
|
|
when the init script is symlinked to a runlevel (e.g. ``S02celeryd``).
|
|
(Issue #1740).
|
|
|
|
This also removed a rarely used feature where you can symlink the script
|
|
to provide alternative configurations. You instead copy the script
|
|
and give it a new name, but perhaps a better solution is to provide
|
|
arguments to ``CELERYD_OPTS`` to separate them:
|
|
|
|
.. code-block:: bash
|
|
|
|
CELERYD_NODES="X1 X2 Y1 Y2"
|
|
CELERYD_OPTS="-A:X1 x -A:X2 x -A:Y1 y -A:Y2 y"
|
|
|
|
- Fallback chord unlock task is now always called after the chord header
|
|
(Issue #1700).
|
|
|
|
This means that the unlock task will not be started if there's
|
|
an error sending the header.
|
|
|
|
- Celery command: Fixed problem with arguments for some control commands.
|
|
|
|
Fix contributed by Konstantin Podshumok.
|
|
|
|
- Fixed bug in ``utcoffset`` where the offset when in DST would be
|
|
completely wrong (Issue #1743).
|
|
|
|
- Worker: Errors occurring while attempting to serialize the result of a
|
|
task will now cause the task to be marked with failure and a
|
|
:class:`kombu.exceptions.EncodingError` error.
|
|
|
|
Fix contributed by Ionel Cristian Mărieș.
|
|
|
|
- Worker with ``-B`` argument did not properly shut down the beat instance.
|
|
|
|
- Worker: The ``%n`` and ``%h`` formats are now also supported by the
|
|
:option:`--logfile`, :option:`--pidfile` and :option:`--statedb` arguments.
|
|
|
|
Example:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ celery -A proj worker -n foo@%h --logfile=%n.log --statedb=%n.db
|
|
|
|
- Redis/Cache result backends: Will now timeout if keys evicted while trying
|
|
to join a chord.
|
|
|
|
- The fallbock unlock chord task now raises :exc:`Retry` so that the
|
|
retry even is properly logged by the worker.
|
|
|
|
- Multi: Will no longer apply Eventlet/gevent monkey patches (Issue #1717).
|
|
|
|
- Redis result backend: Now supports UNIX sockets.
|
|
|
|
Like the Redis broker transport the result backend now also supports
|
|
using ``redis+socket:///tmp/redis.sock`` URLs.
|
|
|
|
Contributed by Alcides Viamontes Esquivel.
|
|
|
|
- Events: Events sent by clients was mistaken for worker related events
|
|
(Issue #1714).
|
|
|
|
For ``events.State`` the tasks now have a ``Task.client`` attribute
|
|
that is set when a ``task-sent`` event is being received.
|
|
|
|
Also, a clients logical clock is not in sync with the cluster so
|
|
they live in a "time bubble". So for this reason monitors will no
|
|
longer attempt to merge with the clock of an event sent by a client,
|
|
instead it will fake the value by using the current clock with
|
|
a skew of -1.
|
|
|
|
- Prefork pool: The method used to find terminated processes was flawed
|
|
in that it did not also take into account missing popen objects.
|
|
|
|
- Canvas: ``group`` and ``chord`` now works with anon signatures as long
|
|
as the group/chord object is associated with an app instance (Issue #1744).
|
|
|
|
You can pass the app by using ``group(..., app=app)``.
|
|
|
|
.. _version-3.1.6:
|
|
|
|
3.1.6
|
|
=====
|
|
:release-date: 2013-12-02 06:00 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- Now depends on :mod:`billiard` 3.3.0.10.
|
|
|
|
- Now depends on :ref:`Kombu 3.0.7 <kombu:version-3.0.7>`.
|
|
|
|
- Fixed problem where Mingle caused the worker to hang at startup
|
|
(Issue #1686).
|
|
|
|
- Beat: Would attempt to drop privileges twice (Issue #1708).
|
|
|
|
- Windows: Fixed error with ``geteuid`` not being available (Issue #1676).
|
|
|
|
- Tasks can now provide a list of expected error classes (Issue #1682).
|
|
|
|
The list should only include errors that the task is expected to raise
|
|
during normal operation::
|
|
|
|
@task(throws=(KeyError, HttpNotFound))
|
|
|
|
What happens when an exceptions is raised depends on the type of error:
|
|
|
|
- Expected errors (included in ``Task.throws``)
|
|
|
|
Will be logged using severity ``INFO``, and traceback is excluded.
|
|
|
|
- Unexpected errors
|
|
|
|
Will be logged using severity ``ERROR``, with traceback included.
|
|
|
|
- Cache result backend now compatible with Python 3 (Issue #1697).
|
|
|
|
- CentOS init script: Now compatible with sys-v style init symlinks.
|
|
|
|
Fix contributed by Jonathan Jordan.
|
|
|
|
- Events: Fixed problem when task name is not defined (Issue #1710).
|
|
|
|
Fix contributed by Mher Movsisyan.
|
|
|
|
- Task: Fixed unbound local errors (Issue #1684).
|
|
|
|
Fix contributed by Markus Ullmann.
|
|
|
|
- Canvas: Now unrolls groups with only one task (optimization) (Issue #1656).
|
|
|
|
- Task: Fixed problem with eta and timezones.
|
|
|
|
Fix contributed by Alexander Koval.
|
|
|
|
- Django: Worker now performs model validation (Issue #1681).
|
|
|
|
- Task decorator now emits less confusing errors when used with
|
|
incorrect arguments (Issue #1692).
|
|
|
|
- Task: New method ``Task.send_event`` can be used to send custom events
|
|
to Flower and other monitors.
|
|
|
|
- Fixed a compatibility issue with non-abstract task classes
|
|
|
|
- Events from clients now uses new node name format (``gen<pid>@<hostname>``).
|
|
|
|
- Fixed rare bug with Callable not being defined at interpreter shutdown
|
|
(Issue #1678).
|
|
|
|
Fix contributed by Nick Johnson.
|
|
|
|
- Fixed Python 2.6 compatibility (Issue #1679).
|
|
|
|
.. _version-3.1.5:
|
|
|
|
3.1.5
|
|
=====
|
|
:release-date: 2013-11-21 06:20 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- Now depends on :ref:`Kombu 3.0.6 <kombu:version-3.0.6>`.
|
|
|
|
- Now depends on :mod:`billiard` 3.3.0.8
|
|
|
|
- App: ``config_from_object`` is now lazy (Issue #1665).
|
|
|
|
- App: ``autodiscover_tasks`` is now lazy.
|
|
|
|
Django users should now wrap access to the settings object
|
|
in a lambda::
|
|
|
|
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
|
|
|
|
this ensures that the settings object is not prepared
|
|
prematurely.
|
|
|
|
- Fixed regression for ``--app`` argument experienced by
|
|
some users (Issue #1653).
|
|
|
|
- Worker: Now respects the ``--uid`` and ``--gid`` arguments
|
|
even if ``--detach`` is not enabled.
|
|
|
|
- Beat: Now respects the ``--uid`` and ``--gid`` arguments
|
|
even if ``--detach`` is not enabled.
|
|
|
|
- Python 3: Fixed unorderable error occuring with the worker ``-B``
|
|
argument enabled.
|
|
|
|
- ``celery.VERSION`` is now a named tuple.
|
|
|
|
- ``maybe_signature(list)`` is now applied recursively (Issue #1645).
|
|
|
|
- ``celery shell`` command: Fixed ``IPython.frontend`` deprecation warning.
|
|
|
|
- The default app no longer includes the builtin fixups.
|
|
|
|
This fixes a bug where ``celery multi`` would attempt
|
|
to load the Django settings module before entering
|
|
the target working directory.
|
|
|
|
- The Django daemonization tutorial was changed.
|
|
|
|
Users no longer have to explicitly export ``DJANGO_SETTINGS_MODULE``
|
|
in :file:`/etc/default/celeryd` when the new project layout is used.
|
|
|
|
- Redis result backend: expiry value can now be 0 (Issue #1661).
|
|
|
|
- Censoring settings now accounts for non-string keys (Issue #1663).
|
|
|
|
- App: New ``autofinalize`` option.
|
|
|
|
Apps are automatically finalized when the task registry is accessed.
|
|
You can now disable this behavior so that an exception is raised
|
|
instead.
|
|
|
|
Example:
|
|
|
|
.. code-block:: python
|
|
|
|
app = Celery(autofinalize=False)
|
|
|
|
# raises RuntimeError
|
|
tasks = app.tasks
|
|
|
|
@app.task
|
|
def add(x, y):
|
|
return x + y
|
|
|
|
# raises RuntimeError
|
|
add.delay(2, 2)
|
|
|
|
app.finalize()
|
|
# no longer raises:
|
|
tasks = app.tasks
|
|
add.delay(2, 2)
|
|
|
|
- The worker did not send monitoring events during shutdown.
|
|
|
|
- Worker: Mingle and gossip is now automatically disabled when
|
|
used with an unsupported transport (Issue #1664).
|
|
|
|
- ``celery`` command: Preload options now supports
|
|
the rare ``--opt value`` format (Issue #1668).
|
|
|
|
- ``celery`` command: Accidentally removed options
|
|
appearing before the subcommand, these are now moved to the end
|
|
instead.
|
|
|
|
- Worker now properly responds to ``inspect stats`` commands
|
|
even if received before startup is complete (Issue #1659).
|
|
|
|
- :signal:`task_postrun` is now sent within a finally block, to make
|
|
sure the signal is always sent.
|
|
|
|
- Beat: Fixed syntax error in string formatting.
|
|
|
|
Contributed by nadad.
|
|
|
|
- Fixed typos in the documentation.
|
|
|
|
Fixes contributed by Loic Bistuer, sunfinite.
|
|
|
|
- Nested chains now works properly when constructed using the
|
|
``chain`` type instead of the ``|`` operator (Issue #1656).
|
|
|
|
.. _version-3.1.4:
|
|
|
|
3.1.4
|
|
=====
|
|
:release-date: 2013-11-15 11:40 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- Now depends on :ref:`Kombu 3.0.5 <kombu:version-3.0.5>`.
|
|
|
|
- Now depends on :mod:`billiard` 3.3.0.7
|
|
|
|
- Worker accidentally set a default socket timeout of 5 seconds.
|
|
|
|
- Django: Fixup now sets the default app so that threads will use
|
|
the same app instance (e.g. for manage.py runserver).
|
|
|
|
- Worker: Fixed Unicode error crash at startup experienced by some users.
|
|
|
|
- Calling ``.apply_async`` on an empty chain now works again (Issue #1650).
|
|
|
|
- The ``celery multi show`` command now generates the same arguments
|
|
as the start command does.
|
|
|
|
- The ``--app`` argument could end up using a module object instead
|
|
of an app instance (with a resulting crash).
|
|
|
|
- Fixed a syntax error problem in the celerybeat init script.
|
|
|
|
Fix contributed by Vsevolod.
|
|
|
|
- Tests now passing on PyPy 2.1 and 2.2.
|
|
|
|
.. _version-3.1.3:
|
|
|
|
3.1.3
|
|
=====
|
|
:release-date: 2013-11-13 00:55 A.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- Fixed compatibility problem with Python 2.7.0 - 2.7.5 (Issue #1637)
|
|
|
|
``unpack_from`` started supporting ``memoryview`` arguments
|
|
in Python 2.7.6.
|
|
|
|
- Worker: :option:`-B` argument accidentally closed files used
|
|
for logging.
|
|
|
|
- Task decorated tasks now keep their docstring (Issue #1636)
|
|
|
|
.. _version-3.1.2:
|
|
|
|
3.1.2
|
|
=====
|
|
:release-date: 2013-11-12 08:00 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- Now depends on :mod:`billiard` 3.3.0.6
|
|
|
|
- No longer needs the billiard C extension to be installed.
|
|
|
|
- The worker silently ignored task errors.
|
|
|
|
- Django: Fixed ``ImproperlyConfigured`` error raised
|
|
when no database backend specified.
|
|
|
|
Fix contributed by j0hnsmith
|
|
|
|
- Prefork pool: Now using ``_multiprocessing.read`` with ``memoryview``
|
|
if available.
|
|
|
|
- ``close_open_fds`` now uses ``os.closerange`` if available.
|
|
|
|
- ``get_fdmax`` now takes value from ``sysconfig`` if possible.
|
|
|
|
.. _version-3.1.1:
|
|
|
|
3.1.1
|
|
=====
|
|
:release-date: 2013-11-11 06:30 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
- Now depends on :mod:`billiard` 3.3.0.4.
|
|
|
|
- Python 3: Fixed compatibility issues.
|
|
|
|
- Windows: Accidentally showed warning that the billiard C extension
|
|
was not installed (Issue #1630).
|
|
|
|
- Django: Tutorial updated with a solution that sets a default
|
|
:envvar:`DJANGO_SETTINGS_MODULE` so that it doesn't have to be typed
|
|
in with the :program:`celery` command.
|
|
|
|
Also fixed typos in the tutorial, and added the settings
|
|
required to use the Django database backend.
|
|
|
|
Thanks to Chris Ward, orarbel.
|
|
|
|
- Django: Fixed a problem when using the Django settings in Django 1.6.
|
|
|
|
- Django: Fixup should not be applied if the django loader is active.
|
|
|
|
- Worker: Fixed attribute error for ``human_write_stats`` when using the
|
|
compatibility prefork pool implementation.
|
|
|
|
- Worker: Fixed compatibility with billiard without C extension.
|
|
|
|
- Inspect.conf: Now supports a ``with_defaults`` argument.
|
|
|
|
- Group.restore: The backend argument was not respected.
|
|
|
|
.. _version-3.1.0:
|
|
|
|
3.1.0
|
|
=======
|
|
:release-date: 2013-11-09 11:00 P.M UTC
|
|
:release-by: Ask Solem
|
|
|
|
See :ref:`whatsnew-3.1`.
|