Installation Guide#

To install a binary distribution, the only requirement is to have a working (recent, 3.7.3 and onwards) version of Python in your system. If you don’t have Python yet you can download it here.

Shapelets Installation#

Binary Installation#

PIP#

To install Shapelets using pip, you can install with:

>>> pip install shapelets-platform

Note

It’s a good idea to use a virtual environment or a docker to avoid conflicts between libraries installed on your system. If you are not familiar with these methods, you can find instructions below.

If you want want to use the Virtual File System (VFS) feature, you need to install the drivers. You can do it with:

>>> pip install shapelets-platform[vfs-all]

Alert: With the all option, you will install all the drivers. We currently support Azure Blob Storage Gen1 & Gen2, Server Message Block (SMB) and the local filesystem. Other fsspec-compatible protocols will be available soon.

To check if Shapelets is installed, you can execute:

>>> python -c "import shapelets as sh; print(sh.__version__)"

It’s a good idea to use a virtual environment or docker image, to avoid conflicts between versions.

Conda#

Currently, conda installation is not available.

Virtual environment#

If you want to install shapelets on a safe environment, you could also create a virtual environment that will help you to safely install shapelets without risking any corruptions. I will show you in the following lines how to do this.

First you want to create a directory

>>> mkdir shapelets_ws

Then you will change directory

>>> cd shapelets_ws

And you will create the virtual environment in the directory

>>> virtualenv venv

Afterward you will activate the virtual environment

>>> source ./venv/bin/activate

Finaly you can use the pip install command to install shapelets in this environment.

Docker#

If you want to install Shapelets on a Docker container, we recommend you use our Docker Image, you can download it with:

>>> docker pull shapeletsdev/shapelets-platform:latest

If you wish to install Shapelets on a Docker container, we recommend using our Docker image. You can download it with:

>>> docker run --name shapelets -p 4567:4567 -d -v /[...]:/io shapeletsdev/shapelets-platform:latest

[…] = insert your own directory

This Docker image needs to expose the 4567 port, and we recommend that you create a volume to share your local scripts with the Docker container.

Now open your web browser and try http://localhost:4567

Recommendations#

Once you install Shapelets, you are ready to use it. If you want a recommendation about development environments, we recommend you:

  • To write and execute code interactively for exploratory data analysis and processing, use notebooks in JupyterLab.

  • To write large scripts for processing data or build DataApps use an IDE, like Visual Studio Code.

How to run Shapelets server#

Once you have installed shapelets, in order to get started you need user credentials to access the UI, and you need to start the server.

You can create a new user using the built-in shapelets tool. Let’s create a user with username johndoe and password passwd.

>>> shapelets client --local --usr johndoe --pwd passwd --create

Now, in order to execute the server, run the following command. We will add the --daemon parameter to keep the server running in the background.

>>> shapelets server run --daemon

Now you can go to localhost and login.

You can know more about this built-in tool in Configuration wizard.

Note

When using Shapelets’ docker image, the server runs automatically upon docker start and a default user, with username admin and password admin is available by default.

Configuration wizard#

Shapelets have a toml configuration file called settings.toml. This configuration file have the params to configure the shapelets server.

The shapelets software have a configuration wizard to configure this file. You can execute this wizard by:

>>> shapelets -h
usage: shapelets [-h] {server,client} ...

Shapelets Platform Tools

positional arguments:
{server,client}
    server         Server tools
    client         Client Defaults

optional arguments:
-h, --help       show this help message and exit

Server#

Shapelets includes a wizard to manage all aspects related to the shapelets server, such as its configuration or its correct execution. You can execute the shapelets server configuration using:

>>> shapelets server -h
usage: shapelets server [-h] {show,config,run,systemd} ...

positional arguments:
{show,config,run,systemd}
    show                Shows current configuration
    config              Configures server settings
    run                 Runs the server
    systemd             Generates daemon configuration for systemd

optional arguments:
-h, --help            show this help message and exit

What do you can do with this wizard? Check this out!

Check the actual config#

You can check the actual settings using:

>>> usage: shapelets server show [-h] [--plain-passwords] path

positional arguments:
path               Working folder where the settings will be loaded from

optional arguments:
-h, --help         show this help message and exit
--plain-passwords  Show plain passwords

For example, checking the default settings:

>>> shapelets server show .
[server]
host = "0.0.0.0"
port = 4567
static = "/home/shapelets/lib/python3.8/site-packages/shapelets/svr/www"
loop = "auto"
secret = "**********"
salt = "csAz7ZXZHdLYQZcEpR6rcg=="

[server.http]
protocol = "auto"

[server.ws]
protocol = "auto"

[server.reload]
enabled = false

[database]
path = "~/.shapelets/data.db"
db_schema = "shapelets"

Modify the config#

The shapelets configuration takes into account a lot of important aspects for a correct and safe deployment, you can see here:

>>> shapelets server config -h
usage: shapelets server config [-h] [--host HOST] [--port PORT] [--uds UDS] [--fd FD] [--limit-concurrency LIMIT_CONCURRENCY]
                            [--backlog BACKLOG] [--timeout-keep-alive TIMEOUT_KEEP_ALIVE] [--ssl-keyfile SSL_KEYFILE]
                            [--ssl-certfile SSL_CERTFILE] [--ssl-keyfile-password SSL_KEYFILE_PASSWORD] [--ssl-version SSL_VERSION]
                            [--ssl-cert-reqs SSL_CERT_REQS] [--ssl-ca-certs SSL_CA_CERTS] [--ssl-ciphers SSL_CIPHERS]
                            [--database-path DATABASE_PATH] [--database-temp-directory DATABASE_TEMP_DIRECTORY]
                            [--database-threads DATABASE_THREADS] [--database-memory-limit DATABASE_MEMORY_LIMIT]
                            destination

Creates a configuration file for a server

positional arguments:
destination           Folder where the settings will be saved

optional arguments:
-h, --help            show this help message and exit
--host HOST           Address to bind the server
--port PORT           Port number of the server
--uds UDS             Unix domain socket
--fd FD               Bind to socket from this file descriptor
--limit-concurrency LIMIT_CONCURRENCY
                        Maximum number of concurrent connections or tasks to allow, before issuing  HTTP 503 responses.
--backlog BACKLOG     Maximum number of connections to hold in backlog.
--timeout-keep-alive TIMEOUT_KEEP_ALIVE
                        Close Keep-Alive connections if no new data is received within this timeout.
--ssl-keyfile SSL_KEYFILE
                        SSL key file
--ssl-certfile SSL_CERTFILE
                        SSL certificate file
--ssl-keyfile-password SSL_KEYFILE_PASSWORD
                        SSL keyfile password
--ssl-version SSL_VERSION
                        SSL version to use
--ssl-cert-reqs SSL_CERT_REQS
                        Whether client certificate is required
--ssl-ca-certs SSL_CA_CERTS
                        CA certificates file
--ssl-ciphers SSL_CIPHERS
                        Ciphers to use
--database-path DATABASE_PATH
                        Path to the file containing the configuration database
--database-temp-directory DATABASE_TEMP_DIRECTORY
                        Temporal directory for result handling and buffering
--database-threads DATABASE_THREADS
                        Maximum number of threads to use in data processing operations
--database-memory-limit DATABASE_MEMORY_LIMIT
                        Maximum memory to use during memory operations.

These settings may be also loaded from the environment, using the same
names of these settings with `shapelets_server` prefix.

For example, if host is not provided by the configuration file,
it will try to resolve its value by reading an environment variable
called `shapelets_server__host` (dashes are converted to
underscores).

Database settings can also be set through environment settings; for
each of the options offered, the prefix will be `shapelets_database__`,
followed with the name of the property (without `database`).  For
example, `--database-memory-limit` will be looked up in the environment
as `shapelets_database__memory_limit`.

Environment values have lower priority over settings found in
configuration files.

Run the server#

To run the server you can execute run command and server will start with the settings in settings.toml file:

>>> shapelets server run
INFO:     Started server process [1653903]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:4567 (Press CTRL+C to quit)

In addition, you can launch the shapelets server in daemon mode, so that the server will run even if you close your terminal.

>>> shapelets server run --daemon
Running Shapelets as a daemon:
    Lock file: /home/shapelets/shapelets.lock
    Log file : /home/shapelets/shapelets.log
    Pid file : /home/shapelets/shapelets.pid
To stop it:

cat /home/shapelets/shapelets.pid | xargs kill

Create a unix service#

Shapelets has a wizard that allows you to quickly set up a unix service to deploy the server.

It also shows you the exact commands to install the service without any problems. You can run it by launching:

>>> shapelets server systemd -h
usage: shapelets server systemd [-h] [-n NAME] [-w WRK] [-u USR] [-g GRP] [-d DST]

Generates a service template that runs a shapelets server using systemd.

optional arguments:
-h, --help            show this help message and exit
-n NAME, --name NAME  Name of the service template
-w WRK, --wrk WRK     Working directory
-u USR, --usr USR     User running the server
-g GRP, --grp GRP     Group to run the server
-d DST, --dst DST     Folder to store the service template (/etc/systemd/system/).

Creating service templates may require sudo permissions, since
it will try to write the template to /etc/systemd/system/

You may change the template destination using argument -d, providing
a folder where to store the template.

To start a server
-----------------
If the service has been named `shapelets`:

    systemctl enable shapelets
    systemctl daemon-reload
    systemctl start shapelets

To query a server
-----------------
    systemctl status shapelets

To stop a server
----------------
    systemctl stop shapelets

To remove a server
------------------
    systemctl disable shapelets
    rm /etc/systemd/system/shapelets.service
    rm /usr/lib/systemd/system/shapelets.service
    systemctl daemon-reload
    systemctl reset-failed

Client#

Shapelets has a complete logging system, which allows you to connect from the same machine to different instances of Shapelets. The shapelets client wizard allows you to interact with these configuration features.

You can execute the shapelet client configuration using:

>>> shapelets client -h
usage: shapelets client [-h] (--local | --url URL) [-u USR]
                    [-p PWD] [--create] [--remember-me]
                    [--enable-telemetry] [--disable-telemetry]

optional arguments:
-h, --help           show this help message and exit
--local              Use Local server
--url URL            URL of the server to connect to
-u USR, --usr USR    User Name
-p PWD, --pwd PWD    User Password
--create             Create user if not exists
--remember-me        Logon and remember credentials
--enable-telemetry   Enable telemetry
--disable-telemetry  Disable telemetry

What do you can do with this wizard? Check this out!

Create a new user#

In your local machine.

>>> shapelets client --local -u new_user -p new_user_pass --create

In a remote server

>>> shapelets client --url my_server_remote:myport -u new_user -p new_user_pass --create

Configure the autologin#

In your local machine.

>>> shapelets client --local -u new_user -p new_user_pass --remember-me

In a remote server

>>> shapelets client --url my_server_remote:myport -u new_user -p new_user_pass --remember-me