add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
#! /usr/bin/python3
|
|
|
|
|
#
|
2021-01-30 20:55:55 +01:00
|
|
|
|
# eoptasks - run commands on servers
|
|
|
|
|
# Copyright (C) 2018-2021 Entr’ouvert
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
#
|
2021-01-30 20:55:55 +01:00
|
|
|
|
# This program is free software: you can redistribute it and/or modify
|
|
|
|
|
# it under the terms of the GNU General Public License as published by
|
|
|
|
|
# the Free Software Foundation, either version 3 of the License, or
|
|
|
|
|
# (at your option) any later version.
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
#
|
2021-01-30 20:55:55 +01:00
|
|
|
|
# This program is distributed in the hope that it will be useful,
|
|
|
|
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
# GNU General Public License for more details.
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
#
|
2021-01-30 20:55:55 +01:00
|
|
|
|
# You should have received a copy of the GNU General Public License
|
|
|
|
|
# along with this program. If not, see <https://www.gnu.org/licenses/>.
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
|
|
|
|
|
import argparse
|
2018-12-12 08:28:44 +01:00
|
|
|
|
import configparser
|
2021-06-05 21:45:19 +02:00
|
|
|
|
import copy
|
2018-12-22 11:31:42 +01:00
|
|
|
|
import curses
|
|
|
|
|
import json
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
import os
|
|
|
|
|
import random
|
|
|
|
|
import re
|
2018-12-22 11:31:42 +01:00
|
|
|
|
import socket
|
2023-02-07 20:08:46 +01:00
|
|
|
|
import subprocess
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
import sys
|
|
|
|
|
import time
|
|
|
|
|
|
|
|
|
|
import libtmux
|
|
|
|
|
import yaml
|
|
|
|
|
|
2020-06-12 17:03:49 +02:00
|
|
|
|
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
class Server:
|
2020-06-12 17:03:49 +02:00
|
|
|
|
def __init__(self, servername, tags=[], display_name=''):
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
self.name = servername
|
2020-07-31 22:26:21 +02:00
|
|
|
|
self.display_name = display_name or self.name
|
2020-06-12 17:03:49 +02:00
|
|
|
|
self.keywords = set(re.split(r'[-_ \.]', servername))
|
|
|
|
|
for tag in tags:
|
|
|
|
|
self.keywords.add(tag)
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
|
2018-12-14 18:48:48 +01:00
|
|
|
|
# add all possible hostname parts as keywords,
|
|
|
|
|
# ex: node1.dev.entrouvert.org will add:
|
|
|
|
|
# node1.dev, node1.dev.entrouvert, node1.dev.entrouvert.org,
|
|
|
|
|
# dev.entrouvert, dev.entrouvert.org, entrouvert.org
|
|
|
|
|
parts = servername.split('.')
|
|
|
|
|
for i in range(len(parts) - 1):
|
|
|
|
|
for j in range(i, len(parts)):
|
|
|
|
|
if i != j:
|
|
|
|
|
self.keywords.add('.'.join(parts[i : j + 1]))
|
2020-12-30 12:14:02 +01:00
|
|
|
|
if i == 0:
|
|
|
|
|
# add first component without trailing digits, this allows
|
|
|
|
|
# matching db1.prod.saas.entrouvert.org with the db
|
|
|
|
|
# keyword.
|
|
|
|
|
self.keywords.add(re.sub(r'\d+$', '', parts[0]))
|
2018-12-14 18:48:48 +01:00
|
|
|
|
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
def __repr__(self):
|
|
|
|
|
return '<Server %s %r>' % (self.name, self.keywords)
|
|
|
|
|
|
|
|
|
|
|
2021-04-13 12:25:34 +02:00
|
|
|
|
def get_config():
|
2018-12-22 10:35:04 +01:00
|
|
|
|
config = configparser.ConfigParser()
|
|
|
|
|
config.read(os.path.join(os.path.expanduser('~/.config/eoptasks.ini')))
|
2021-04-13 12:25:34 +02:00
|
|
|
|
return config
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def get_servers():
|
|
|
|
|
servers = []
|
|
|
|
|
config = get_config()
|
2020-06-12 17:03:49 +02:00
|
|
|
|
serversfile = config.get('config', 'servers', fallback=None)
|
|
|
|
|
if serversfile is None:
|
2018-12-22 10:35:04 +01:00
|
|
|
|
print(
|
|
|
|
|
'You need to create ~/.config/eoptasks.ini with such a content:\n'
|
|
|
|
|
'\n'
|
|
|
|
|
' [config]\n'
|
2020-06-12 17:03:49 +02:00
|
|
|
|
' servers = /home/user/src/puppet/data/servers.yaml\n'
|
|
|
|
|
)
|
2018-12-22 10:35:04 +01:00
|
|
|
|
sys.exit(1)
|
2018-12-22 14:58:51 +01:00
|
|
|
|
ignorelist = [x.strip() for x in config.get('config', 'ignore', fallback='').split(',')]
|
2018-12-22 10:35:04 +01:00
|
|
|
|
|
2020-09-09 08:43:15 +02:00
|
|
|
|
stripsuffixes = [x.strip() for x in config.get('config', 'stripsuffix', fallback='').split(',')]
|
2020-07-31 22:26:21 +02:00
|
|
|
|
|
|
|
|
|
def get_display_name(x):
|
2020-09-09 08:43:15 +02:00
|
|
|
|
for stripsuffix in stripsuffixes:
|
|
|
|
|
if stripsuffix and x.endswith(stripsuffix):
|
|
|
|
|
return x[: -len(stripsuffix)]
|
2020-07-31 22:26:21 +02:00
|
|
|
|
return x
|
|
|
|
|
|
2021-02-06 14:12:57 +01:00
|
|
|
|
# load servers from servers.yaml
|
2020-06-12 17:03:49 +02:00
|
|
|
|
for s in yaml.safe_load(open(serversfile))['servers']:
|
|
|
|
|
servername, tags = s.get('name'), s.get('tags', [])
|
|
|
|
|
if servername in ignorelist:
|
|
|
|
|
continue
|
|
|
|
|
servers.append(Server(servername, tags, display_name=get_display_name(servername)))
|
|
|
|
|
|
2021-02-06 14:12:57 +01:00
|
|
|
|
# load additional servers from eoptasks.ini
|
|
|
|
|
if config.has_section('servers'):
|
|
|
|
|
for server in config.options('servers'):
|
|
|
|
|
servername = server
|
|
|
|
|
tags = [x.strip() for x in config.get('servers', server).split(',')]
|
|
|
|
|
servers.append(Server(servername, tags, display_name=get_display_name(servername)))
|
|
|
|
|
|
2021-06-05 21:45:55 +02:00
|
|
|
|
for server in servers:
|
|
|
|
|
if config.has_section('server:%s' % server.name):
|
|
|
|
|
# server.key_automation will be a list
|
|
|
|
|
# [content to expect, list of keys to send]
|
|
|
|
|
server.key_automation = []
|
|
|
|
|
for key, value in sorted(config.items('server:%s' % server.name)):
|
|
|
|
|
if key.startswith('expect'):
|
|
|
|
|
server.key_automation.append([value, []])
|
|
|
|
|
elif key.startswith('send'):
|
|
|
|
|
server.key_automation[-1][1].append(value)
|
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
return servers
|
|
|
|
|
|
2020-06-12 17:03:49 +02:00
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
def parse_args():
|
|
|
|
|
parser = argparse.ArgumentParser()
|
|
|
|
|
parser.add_argument('-l', '--list-servers', action='store_true')
|
2018-12-31 15:25:37 +01:00
|
|
|
|
parser.add_argument('--session-name', dest='session_name', type=str)
|
|
|
|
|
parser.add_argument('--status-window', action='store_true')
|
2018-12-31 16:31:29 +01:00
|
|
|
|
parser.add_argument('--command-window', action='store_true')
|
|
|
|
|
parser.add_argument('--command-server-name', dest='command_server_name', type=str)
|
2019-01-05 14:37:17 +01:00
|
|
|
|
parser.add_argument('--noinput', dest='noinput', action='store_true')
|
2021-01-23 19:11:32 +01:00
|
|
|
|
parser.add_argument('--list-commands', dest='list_commands', action='store_true')
|
2018-12-22 10:35:04 +01:00
|
|
|
|
parser.add_argument('-k', dest='keywords', type=str)
|
2020-06-26 16:41:52 +02:00
|
|
|
|
parser.add_argument('-x', dest='exclude', type=str)
|
2018-12-22 10:35:04 +01:00
|
|
|
|
parser.add_argument('cmd', type=str, nargs='?', default=None)
|
2018-12-29 21:07:08 +01:00
|
|
|
|
parser.add_argument('args', nargs=argparse.REMAINDER)
|
2018-12-22 10:35:04 +01:00
|
|
|
|
args = parser.parse_args()
|
|
|
|
|
return args
|
|
|
|
|
|
2021-01-30 20:47:28 +01:00
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
def filter_servers(servers, args):
|
|
|
|
|
selected_servers = []
|
|
|
|
|
if args.keywords:
|
2021-04-13 12:25:47 +02:00
|
|
|
|
config = get_config()
|
2021-05-12 08:49:22 +02:00
|
|
|
|
cmd_keywords = args.keywords
|
|
|
|
|
if config.has_section('keywords'):
|
|
|
|
|
cmd_keywords = config['keywords'].get(args.keywords, args.keywords)
|
2021-04-13 12:25:47 +02:00
|
|
|
|
|
|
|
|
|
for keyword in cmd_keywords.split(','):
|
2018-12-22 10:35:04 +01:00
|
|
|
|
keywords = set(keyword.split('/'))
|
|
|
|
|
selected_servers.extend(
|
|
|
|
|
[x for x in servers if keywords.issubset(x.keywords) and not x in selected_servers]
|
|
|
|
|
)
|
2021-04-13 12:25:47 +02:00
|
|
|
|
for keyword in cmd_keywords.split(','):
|
2018-12-22 10:35:04 +01:00
|
|
|
|
if keyword.startswith('!') or keyword.startswith('-'):
|
|
|
|
|
selected_servers = [x for x in selected_servers if keyword[1:] not in x.keywords]
|
|
|
|
|
else:
|
|
|
|
|
selected_servers = servers
|
2020-06-26 16:41:52 +02:00
|
|
|
|
if args.exclude:
|
|
|
|
|
for exclude in args.exclude.split(','):
|
|
|
|
|
regex = re.compile(exclude)
|
|
|
|
|
selected_servers = [x for x in selected_servers if not regex.match(x.name)]
|
2018-12-22 10:35:04 +01:00
|
|
|
|
return selected_servers
|
|
|
|
|
|
2020-06-26 16:41:52 +02:00
|
|
|
|
|
2018-12-31 16:31:29 +01:00
|
|
|
|
def status_window(args):
|
|
|
|
|
session_name = args.session_name
|
2018-12-22 11:31:42 +01:00
|
|
|
|
curses.setupterm()
|
|
|
|
|
window = curses.initscr()
|
|
|
|
|
window.addstr(0, 0, 'eoptasks', curses.A_STANDOUT)
|
|
|
|
|
window.addstr(0, 10, '🙂')
|
|
|
|
|
curses.curs_set(0)
|
|
|
|
|
window.refresh()
|
|
|
|
|
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
|
|
|
|
server_address = '/tmp/.eoptasks.%s' % session_name
|
|
|
|
|
sock.bind(server_address)
|
|
|
|
|
sock.listen(1)
|
|
|
|
|
e = None
|
2018-12-31 16:31:29 +01:00
|
|
|
|
servers_results = {}
|
2018-12-22 11:31:42 +01:00
|
|
|
|
while True:
|
|
|
|
|
connection, client_address = sock.accept()
|
|
|
|
|
try:
|
|
|
|
|
json_msg = b''
|
|
|
|
|
while True:
|
|
|
|
|
data = connection.recv(5000)
|
|
|
|
|
if not data:
|
|
|
|
|
break
|
|
|
|
|
json_msg += data
|
2018-12-31 16:31:29 +01:00
|
|
|
|
msg = json.loads(json_msg.decode('utf-8'))
|
2018-12-22 11:31:42 +01:00
|
|
|
|
finally:
|
|
|
|
|
connection.close()
|
|
|
|
|
|
2018-12-31 16:31:29 +01:00
|
|
|
|
if msg.get('@type') == 'servers-info':
|
|
|
|
|
servers_info = msg['info']
|
|
|
|
|
elif msg.get('@type') == 'server-result':
|
|
|
|
|
servers_results.update(msg['info'])
|
|
|
|
|
|
2018-12-22 11:31:42 +01:00
|
|
|
|
try:
|
|
|
|
|
height, width = window.getmaxyx()
|
2020-07-31 22:26:21 +02:00
|
|
|
|
max_length = max([len(x['display_name']) for x in servers_info.values()]) + 4
|
2018-12-22 11:31:42 +01:00
|
|
|
|
nb_columns = (width - 4) // max_length
|
|
|
|
|
|
|
|
|
|
for i, server_name in enumerate(servers_info):
|
|
|
|
|
y = 2 + (i // nb_columns)
|
|
|
|
|
x = 1 + (width // nb_columns) * (i % nb_columns)
|
2020-07-31 22:26:21 +02:00
|
|
|
|
window.addstr(y, x + 3, servers_info[server_name]['display_name'])
|
2018-12-22 11:31:42 +01:00
|
|
|
|
status_icon = {
|
|
|
|
|
'running': '⏳',
|
|
|
|
|
'done': '🆗',
|
|
|
|
|
}.get(servers_info[server_name]['status'], '💤')
|
2018-12-31 16:31:29 +01:00
|
|
|
|
if servers_results.get(server_name) == 'error':
|
|
|
|
|
status_icon = '❗'
|
2018-12-22 11:31:42 +01:00
|
|
|
|
window.addstr(y, x, status_icon)
|
|
|
|
|
if y > height - 4:
|
|
|
|
|
break
|
|
|
|
|
window.refresh()
|
|
|
|
|
total_servers = len(servers_info.keys())
|
|
|
|
|
running_servers = len([x for x in servers_info.values() if x['status'] == 'running'])
|
|
|
|
|
done_servers = len([x for x in servers_info.values() if x['status'] == 'done'])
|
|
|
|
|
if total_servers == done_servers:
|
|
|
|
|
break
|
|
|
|
|
except Exception as e:
|
|
|
|
|
window.addstr(0, 10, '😡 %r' % e)
|
|
|
|
|
window.refresh()
|
|
|
|
|
os.unlink(server_address)
|
|
|
|
|
window.addstr(0, 10, '😎')
|
|
|
|
|
window.refresh()
|
|
|
|
|
time.sleep(5)
|
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
|
2018-12-31 16:31:29 +01:00
|
|
|
|
def send_status_message(tmux_session_name, msg):
|
|
|
|
|
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
|
|
|
|
server_address = '/tmp/.eoptasks.%s' % tmux_session_name
|
|
|
|
|
try:
|
|
|
|
|
sock.connect(server_address)
|
2023-02-07 20:08:46 +01:00
|
|
|
|
except OSError as e:
|
2018-12-31 16:31:29 +01:00
|
|
|
|
return
|
|
|
|
|
sock.sendall(json.dumps(msg).encode('utf-8'))
|
|
|
|
|
sock.close()
|
|
|
|
|
|
|
|
|
|
|
2021-01-23 19:11:32 +01:00
|
|
|
|
def get_commands():
|
|
|
|
|
commands = {
|
2018-12-31 16:31:29 +01:00
|
|
|
|
'apt.update': 'sudo apt update',
|
2023-05-23 19:33:33 +02:00
|
|
|
|
'apt.upgrade': 'sudo apt update && sudo apt full-upgrade -y -o Dpkg::Options::="--force-confold"',
|
2018-12-31 16:31:29 +01:00
|
|
|
|
# collectstatic is useful after an upgrade of gadjo.
|
|
|
|
|
'collectstatic': '''sudo -u authentic-multitenant authentic2-multitenant-manage collectstatic --noinput;
|
|
|
|
|
sudo -u bijoe bijoe-manage collectstatic --noinput;
|
|
|
|
|
sudo -u chrono chrono-manage collectstatic --noinput;
|
|
|
|
|
sudo -u combo combo-manage collectstatic --noinput;
|
|
|
|
|
sudo -u fargo fargo-manage collectstatic --noinput;
|
|
|
|
|
sudo -u hobo hobo-manage collectstatic --noinput;
|
2022-09-13 16:27:32 +02:00
|
|
|
|
sudo -u lingo lingo-manage collectstatic --noinput;
|
2018-12-31 16:31:29 +01:00
|
|
|
|
sudo -u passerelle passerelle-manage collectstatic --noinput;
|
2020-07-16 08:32:49 +02:00
|
|
|
|
sudo -u welco welco-manage collectstatic --noinput;
|
2018-12-31 16:31:29 +01:00
|
|
|
|
sudo -u wcs wcs-manage collectstatic;
|
|
|
|
|
/bin/true'''.replace(
|
|
|
|
|
'\n', ''
|
|
|
|
|
),
|
|
|
|
|
# combo.reload is useful to get a new {% start_timestamp %} after an
|
|
|
|
|
# upgrade of publik-base-theme.
|
|
|
|
|
'combo.reload': '''sudo service combo reload; /bin/true''',
|
|
|
|
|
# hobo-agent.restart is the fastest way to get the number of threads
|
|
|
|
|
# used by celery under control :/
|
2020-05-15 11:21:36 +02:00
|
|
|
|
'hobo-agent.restart': '''test -e /etc/hobo-agent/settings.py && sudo supervisorctl reread && sudo supervisorctl restart hobo-agent''',
|
2019-03-26 16:53:27 +01:00
|
|
|
|
# memcached.restart is useful to force shared theme to be updated.
|
|
|
|
|
'memcached.restart': '''sudo service memcached restart; /bin/true''',
|
2023-05-12 08:37:33 +02:00
|
|
|
|
'restart.all': '''sudo systemctl restart authentic2-multitenant bijoe chrono combo docbow fargo hobo lingo passerelle wcs welco; /bin/true''',
|
|
|
|
|
'reload.all': '''sudo systemctl reload authentic2-multitenant bijoe chrono combo docbow fargo hobo lingo passerelle wcs welco; /bin/true''',
|
2020-06-16 07:49:46 +02:00
|
|
|
|
'passerelle.restart': '''sudo systemctl restart passerelle; /bin/true''',
|
|
|
|
|
'wcs.restart': '''sudo systemctl restart wcs; /bin/true''',
|
2019-12-02 11:29:46 +01:00
|
|
|
|
# puppet.update, unfortunately without proper error checking.
|
|
|
|
|
'puppet.update': '''sudo puppet agent -t || true''',
|
2021-01-22 22:51:21 +01:00
|
|
|
|
}
|
2021-04-13 12:25:34 +02:00
|
|
|
|
config = get_config()
|
2021-01-23 19:11:32 +01:00
|
|
|
|
for section in config.sections():
|
|
|
|
|
if section.startswith('command:'):
|
|
|
|
|
commands[section[len('command:') :]] = config.get(section, 'cmd')
|
|
|
|
|
return commands
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def command_window(args):
|
|
|
|
|
tmux_session_name = args.session_name
|
|
|
|
|
commands = get_commands()
|
|
|
|
|
if args.cmd in commands:
|
|
|
|
|
cmd = commands[args.cmd]
|
2021-01-22 22:51:21 +01:00
|
|
|
|
else:
|
|
|
|
|
cmd = args.cmd
|
|
|
|
|
|
2018-12-31 16:31:29 +01:00
|
|
|
|
if args.args:
|
|
|
|
|
cmd += ' ' + ' '.join(['"%s"' % x for x in args.args])
|
2019-01-02 17:07:46 +01:00
|
|
|
|
orig_cmd = cmd
|
2021-06-05 21:45:19 +02:00
|
|
|
|
|
|
|
|
|
config = get_config()
|
|
|
|
|
environ = {}
|
|
|
|
|
if config.has_section('server:%s' % args.command_server_name):
|
|
|
|
|
new_environ = {}
|
|
|
|
|
for key, value in config.items('server:%s' % args.command_server_name):
|
|
|
|
|
if key.startswith('env'):
|
|
|
|
|
new_environ[value.split('=', 1)[0]] = value.split('=', 1)[1]
|
|
|
|
|
if new_environ:
|
|
|
|
|
environ = copy.copy(os.environ)
|
|
|
|
|
environ.update(new_environ)
|
|
|
|
|
|
|
|
|
|
call_kwargs = {}
|
|
|
|
|
if environ:
|
|
|
|
|
call_kwargs['env'] = environ
|
|
|
|
|
|
2019-01-02 17:07:46 +01:00
|
|
|
|
while True:
|
2019-01-02 18:49:29 +01:00
|
|
|
|
# -t: force a tty for interactive commands.
|
2021-06-05 21:45:19 +02:00
|
|
|
|
rc = subprocess.call(['ssh', '-t', args.command_server_name] + [cmd], **call_kwargs)
|
2019-01-02 17:07:46 +01:00
|
|
|
|
if rc == 0:
|
|
|
|
|
send_status_message(
|
|
|
|
|
tmux_session_name, {'@type': 'server-result', 'info': {args.command_server_name: 'success'}}
|
|
|
|
|
)
|
|
|
|
|
break
|
2018-12-31 16:31:29 +01:00
|
|
|
|
send_status_message(
|
|
|
|
|
tmux_session_name, {'@type': 'server-result', 'info': {args.command_server_name: 'error'}}
|
|
|
|
|
)
|
2019-01-05 14:37:17 +01:00
|
|
|
|
if args.noinput:
|
|
|
|
|
break
|
2019-01-02 17:07:46 +01:00
|
|
|
|
choice = None
|
|
|
|
|
while choice not in ['r', 's', 'q']:
|
|
|
|
|
choice = input('[R]etry, [S]hell, [Q]uit --> ').lower()
|
|
|
|
|
if choice == 'r':
|
|
|
|
|
cmd = orig_cmd
|
|
|
|
|
elif choice == 's':
|
|
|
|
|
cmd = 'bash'
|
|
|
|
|
elif choice == 'q':
|
|
|
|
|
break
|
2018-12-31 16:31:29 +01:00
|
|
|
|
|
2021-01-30 20:47:28 +01:00
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
args = parse_args()
|
|
|
|
|
|
2021-01-23 19:11:32 +01:00
|
|
|
|
if args.list_commands:
|
|
|
|
|
commands = get_commands()
|
|
|
|
|
for command in sorted(commands):
|
|
|
|
|
print(command)
|
|
|
|
|
sys.exit(0)
|
|
|
|
|
|
2018-12-31 16:31:29 +01:00
|
|
|
|
if args.status_window:
|
|
|
|
|
status_window(args)
|
|
|
|
|
sys.exit(0)
|
|
|
|
|
|
|
|
|
|
if args.command_window:
|
|
|
|
|
command_window(args)
|
2018-12-22 10:42:38 +01:00
|
|
|
|
sys.exit(0)
|
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
servers = get_servers()
|
|
|
|
|
selected_servers = filter_servers(servers, args)
|
2023-03-30 14:33:47 +02:00
|
|
|
|
selected_servers = sorted(selected_servers, key=lambda x: x.name)
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
|
2018-12-12 18:52:31 +01:00
|
|
|
|
if args.list_servers:
|
2023-03-30 14:33:47 +02:00
|
|
|
|
for server in selected_servers:
|
2018-12-12 18:52:31 +01:00
|
|
|
|
print(server.name)
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
sys.exit(0)
|
|
|
|
|
|
2018-12-12 18:52:31 +01:00
|
|
|
|
if not selected_servers:
|
|
|
|
|
sys.stderr.write('No matching servers\n')
|
|
|
|
|
sys.exit(1)
|
|
|
|
|
|
|
|
|
|
if not args.cmd:
|
|
|
|
|
sys.stderr.write('Missing command\n')
|
|
|
|
|
sys.exit(1)
|
|
|
|
|
|
2021-01-30 20:47:28 +01:00
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
def init_tmux_session():
|
2019-01-09 17:56:24 +01:00
|
|
|
|
if os.environ.get('TMUX'): # already in a tmux
|
|
|
|
|
sys.stderr.write('Cannot run embedded in tmux\n')
|
|
|
|
|
sys.exit(1)
|
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
tmux_session_name = 's%s' % random.randrange(1000)
|
2018-12-22 11:31:42 +01:00
|
|
|
|
server_address = '/tmp/.eoptasks.%s' % tmux_session_name
|
|
|
|
|
try:
|
|
|
|
|
os.unlink(server_address)
|
|
|
|
|
except OSError:
|
|
|
|
|
pass
|
2018-12-31 16:31:29 +01:00
|
|
|
|
os.environ['SHELL'] = '/bin/sh'
|
2018-12-31 15:25:37 +01:00
|
|
|
|
os.system(
|
|
|
|
|
'tmux new-session -s %s -n 🌑 -d %s --status-window --session-name %s'
|
2018-12-22 10:42:38 +01:00
|
|
|
|
% (tmux_session_name, sys.argv[0], tmux_session_name)
|
2021-01-30 20:47:28 +01:00
|
|
|
|
)
|
2018-12-22 10:35:04 +01:00
|
|
|
|
return tmux_session_name
|
|
|
|
|
|
2021-01-30 20:47:28 +01:00
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
tmux_session_name = init_tmux_session()
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
|
|
|
|
|
pid = os.fork()
|
|
|
|
|
if pid:
|
|
|
|
|
os.system('tmux attach-session -t %s' % tmux_session_name)
|
|
|
|
|
else:
|
2021-01-30 20:47:28 +01:00
|
|
|
|
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
def cluster_name(server_name):
|
2019-01-08 21:42:17 +01:00
|
|
|
|
cluster_name = re.sub(r'\d', '', server_name)
|
|
|
|
|
for location in ('rbx', 'gra', 'sbg'):
|
|
|
|
|
cluster_name = cluster_name.replace('.%s.' % location, '.loc')
|
|
|
|
|
return cluster_name
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
|
2018-12-22 10:35:04 +01:00
|
|
|
|
tmux = libtmux.Server()
|
2023-03-15 14:27:43 +01:00
|
|
|
|
|
|
|
|
|
try:
|
|
|
|
|
session = tmux.sessions.get(session_name=tmux_session_name)
|
|
|
|
|
except AttributeError:
|
|
|
|
|
session = tmux.find_where({'session_name': tmux_session_name})
|
2018-12-22 10:35:04 +01:00
|
|
|
|
|
|
|
|
|
status_window = session.attached_window
|
|
|
|
|
|
2018-12-22 11:31:42 +01:00
|
|
|
|
all_servers = selected_servers[:]
|
2018-12-21 15:15:34 +01:00
|
|
|
|
total_number = len(selected_servers)
|
2018-12-22 11:31:42 +01:00
|
|
|
|
|
|
|
|
|
servers_info = {}
|
|
|
|
|
for server in selected_servers:
|
2020-07-31 22:26:21 +02:00
|
|
|
|
servers_info[server.name] = {'status': '', 'display_name': server.display_name}
|
2018-12-22 11:31:42 +01:00
|
|
|
|
|
|
|
|
|
def send_status():
|
2023-10-06 13:43:58 +02:00
|
|
|
|
current_windows = [x.name for x in session.windows]
|
2018-12-22 11:31:42 +01:00
|
|
|
|
for server in all_servers:
|
|
|
|
|
server_info = servers_info[server.name]
|
|
|
|
|
if server.name in current_windows:
|
|
|
|
|
server_info['status'] = 'running'
|
|
|
|
|
elif server_info['status'] == 'running':
|
|
|
|
|
server_info['status'] = 'done'
|
2018-12-31 16:31:29 +01:00
|
|
|
|
send_status_message(tmux_session_name, {'@type': 'servers-info', 'info': servers_info})
|
2018-12-22 11:31:42 +01:00
|
|
|
|
|
2021-06-05 21:45:55 +02:00
|
|
|
|
def handle_expects():
|
2023-10-06 13:43:58 +02:00
|
|
|
|
for win in session.windows:
|
2021-06-05 21:45:55 +02:00
|
|
|
|
try:
|
|
|
|
|
server = [x for x in all_servers if x.name == win.name][0]
|
|
|
|
|
except IndexError:
|
|
|
|
|
continue
|
|
|
|
|
if not getattr(server, 'key_automation', None):
|
|
|
|
|
continue
|
|
|
|
|
try:
|
|
|
|
|
current_content = win.cmd('capture-pane', '-p').stdout[-1]
|
|
|
|
|
except IndexError:
|
|
|
|
|
pass
|
|
|
|
|
else:
|
|
|
|
|
if current_content.endswith(server.key_automation[0][0]):
|
|
|
|
|
for keys in server.key_automation[0][1]:
|
|
|
|
|
# pane.send_keys sends a leading space character and
|
|
|
|
|
# that makes it unusuable to send a password (for
|
|
|
|
|
# example).
|
|
|
|
|
try:
|
|
|
|
|
win.panes[0].cmd('send-keys', keys + '\n')
|
|
|
|
|
except IndexError:
|
|
|
|
|
continue
|
|
|
|
|
time.sleep(0.1)
|
|
|
|
|
# remove played automation
|
|
|
|
|
server.key_automation = server.key_automation[1:]
|
|
|
|
|
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
while selected_servers:
|
2023-10-06 13:43:58 +02:00
|
|
|
|
current_clusters = [cluster_name(x.name) for x in session.windows]
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
for server in selected_servers[:]:
|
|
|
|
|
if cluster_name(server.name) in current_clusters:
|
|
|
|
|
continue
|
|
|
|
|
selected_servers.remove(server)
|
2019-01-05 14:37:17 +01:00
|
|
|
|
window_cmd = '%s --session-name %s --command-window --command-server-name %s %s "%s" %s' % (
|
2018-12-31 16:31:29 +01:00
|
|
|
|
sys.argv[0],
|
|
|
|
|
tmux_session_name,
|
|
|
|
|
server.name,
|
2019-01-05 14:37:17 +01:00
|
|
|
|
'--noinput' if args.noinput else '',
|
2018-12-31 16:31:29 +01:00
|
|
|
|
args.cmd,
|
|
|
|
|
' '.join(['"%s"' % x for x in args.args]),
|
|
|
|
|
)
|
|
|
|
|
session.new_window(attach=False, window_name=server.name, window_shell=window_cmd)
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
break
|
|
|
|
|
else:
|
|
|
|
|
time.sleep(0.1)
|
2023-10-06 13:43:58 +02:00
|
|
|
|
while len(session.windows) > 10:
|
2018-12-22 11:31:42 +01:00
|
|
|
|
send_status()
|
2021-06-05 21:45:55 +02:00
|
|
|
|
handle_expects()
|
add new script to run commands on servers
This script provides parallel remote execution of commands, while having
some special knownledge of servers that should *not* be handled in parallel.
It defers terminal-handling to tmux(1).
It has some targeting capacities using keywords. Commas for 'OR' and slashes
for 'AND', ex: ext/test,saas/test/passerelle will select all external test
servers + all passerelle servers on the SaaS.
It takes any shell command and has some builtin shortcuts such as apt.update
and apt.upgrade. (that's the whole lot, actually).
Regarding actual performance benefits, apt upgrade with no packages to
upgrade:
$ time eoptasks -k ext/test apt.upgrade
real 0m24,249s
user 0m0,140s
sys 0m0,025s
$ time eotasks -g ext_test apt.upgrade
real 6m9,956s
user 3m32,096s
sys 0m2,322s
2018-12-09 14:12:56 +01:00
|
|
|
|
time.sleep(0.1)
|
2018-12-22 11:31:42 +01:00
|
|
|
|
send_status()
|
2021-06-22 15:34:35 +02:00
|
|
|
|
handle_expects()
|
2018-12-22 11:31:42 +01:00
|
|
|
|
|
2018-12-23 00:47:36 +01:00
|
|
|
|
percentage = (total_number - len(selected_servers)) / total_number
|
2018-12-21 15:15:34 +01:00
|
|
|
|
if percentage == 1:
|
|
|
|
|
status_window.rename_window('🌕')
|
|
|
|
|
elif percentage >= 0.75:
|
|
|
|
|
status_window.rename_window('🌔')
|
|
|
|
|
elif percentage >= 0.5:
|
|
|
|
|
status_window.rename_window('🌓')
|
|
|
|
|
elif percentage >= 0.25:
|
|
|
|
|
status_window.rename_window('🌒')
|
2018-12-22 11:31:42 +01:00
|
|
|
|
|
2023-10-06 13:43:58 +02:00
|
|
|
|
while len(session.windows) > 1:
|
2018-12-22 11:31:42 +01:00
|
|
|
|
send_status()
|
2021-06-05 21:45:55 +02:00
|
|
|
|
handle_expects()
|
2018-12-21 15:15:34 +01:00
|
|
|
|
time.sleep(0.1)
|
2018-12-22 11:31:42 +01:00
|
|
|
|
status_window.rename_window('🌕')
|
|
|
|
|
send_status()
|