The remote client

The “remote client” is a programming library for contacting JuliaBase over the network. Being a library, it is not a program per se but used by your own code for the bidirectional communication with the database. The remote client communicates through a REST HTTP interface implemented in the JuliaBase server. This interface could be used directly, however, the remote client makes it much easier.

In the following, we will describe what the remote client can do and how to get it running. For the API itself, we have to refer to the documentation in the sources of the Python files in remote_client/. Note especially the examples in remote_client/examples/.

Use cases

The remote client has many applications.

Crawlers

Crawlers are programs that go through a bunch of data or log files and transmit their content to the database. For example, a measurement setup may write a data file for every measurement run. Every hour, a crawler is called, which scans for new files. It reads the new files and uses the content to add new measurements processes to the database. This way, new measurements automatically end up in the database. You may have more than one crawler regularly running in your institute.

The very same crawler can also be used to import legacy data. The crawler can be programmed in such a way that during its first run, it imports thousands of data sets of recent years, and from there on, it imports newly added data sets every hour.

Since crawlers run unattendedly, they must be very robust. If a data file is invalid, or if the central server is temporarily unavailable, it must gently exit and try again next time, without forgetting to add something, and without adding things twice.

Moreover, legacy data mostly is poorly structured and organized. If you think your data is the exception, think again. During writing the crawler, you will be surprised how many odd ideas and mistakes your colleagues are capable of.

Be that as it may, crawlers are the primary use case for the remote client. The remote client contains functions especially for making writing them as easy as possible.

Connecting the setup with the database

A more direct way than the crawler for bringing processing and measurement data into the database, albeit not always feasible, is to extend the control program of the experimental setup. For this, you must have access to the source code of the control program. Then, you can use the remote client for writing new runs immediately into the database.

However, the communication needn’t be one-way. You can also use the remote client to authenticate the user, to check whether he or she is allowed to use this setup, and to check whether the sample names belong to valid and for the user accessible samples.

The remote client is written in Python, however, it is easy to use it from programs written in other programming languages, and JuliaBase ships with bindings to Delphi, LabVIEW, and Visual Basic.

Data mining and analysis

Everyone with browser access to the JuliaBase database can use the remote client for accessing, too. The permissions will be the same in both cases, of course. This can be used by researchers capable of programming to get data from the database and do something with it, e.g. creating statistics or evaluating raw data.

Imagine the following: A researcher makes many sample series and measures their temperature-dependent conductivity. The researcher can write a program that extracts the names of the raw data files from the database, then creates a plot for each sample series with the conductivity curves of all samples of that series, and writes these plots back to the database. There, one can see them on the series’ pages, and in every sample’s data sheet.

Extending the remote client

As for JuliaBase itself, also the remote client should be extended with the functionality special to your institution. This is not absolutely necessary, but without it, you can only use the very basic functionality. However, extending the remote client is much easier than extending JuliaBase.

In JuliaBases’ source code, the default remote client resides in the directory remote_client/, and it has the following structure:

remote_client/
    jb_remote_inm.py
    jb_remote/
        __init__.py
        common.py
        samples.py
        settings.py
        ...

Replace the file jb_remote_inm.py with your own institute’s code, and give your file a name derived from your institution’s name. Your file should start with something like:

from jb_remote import *

settings.ROOT_URL = settings.TESTSERVER_ROOT_URL = "https://juliabase.my-institute.edu/"

From there on, you are totally free how to program your incarnation of the remote client. You may use jb_remote_inm.py as a source of inspiration, of course.

Settings

The following settings in the remote client are available and should be set in your jb_remote_inm.py:

ROOT_URL
The URL of the production server. It must end in a slash. Default: None
TESTSERVER_ROOT_URL
The URL of the test server. It must end in a slash. Default: "https://demo.juliabase.org/"
SMTP_SERVER
The DNS name of the SMTP server used for outgoing mail. It may be used in crawlers to send success or error emails. You may add a port number after a colon. Default: "mailrelay.example.com:587"
SMTP_LOGIN
The login name of the SMTP server. If empty, no login is performed. If not empty, TLS is used. Default: "username"
SMTP_PASSWORD
The password used to login to the SMTP server through TLS. Default: "password"
EMAIL_FROM
The sender used for outgoing mails. Default: "me@example.com"
EMAIL_TO
The recipient of outgoing mails. Default: "admins@example.com"

Local installation and usage

Once you finished extending the default remote client, your remote client consists of the files listed above, with jb_remote_inm.py substituted by your file called, say, jb_remote_my_institute.py. This set of files needs to be copied to the machines where the remote client is supposed to be used, and the top directory (the one with your jb_remote_my_institute.py) should be in PYTHONPATH.

Then, you use the remote client with a Python script as easy as this:

from jb_remote_my_institute import *

setup_logging("console")
login("juliabase", "12345")

sample = Sample("14-JS-1")
sample.current_location = "Sean's office"
sample.edit_description = "location changed"
sample.submit()

logout()

By the way, the files are organized in a way that you can update very conveniently: If a new version of JuliaBase is released, you simply have to replace the jb_remote/ subdirectory with the new one.

Classes and functions

As explained in Extending the remote client, the following names become available by saying

from jb_remote import *
connection

The connection to the database server. It is of the type JuliaBaseConnection.

primary_keys

A dict-like object of type PrimaryKeys that is a mapping of identifying keys to IDs. Possible keys are:

"users"
mapping user names to user IDs.
"external_operators"
mapping external operator names to external operator IDs.
"topics"
mapping topic names to topic IDs.
login(username, password, testserver=False)

Logins to JuliaBase.

Parameters:
  • username (str) – the username used to log in
  • password (str) – the user’s password
  • testserver (bool) – whether the testserver should be user. If False, the production server is used.
class JuliaBaseConnection

Class for the routines that connect to the database at HTTP level. This is a singleton class, and its only instance resides at top-level in this module.

open(relative_url, data=None, response_is_json=True)

Do an HTTP request with the JuliaBase server. If data is not None, its a POST request, and GET otherwise.

Parameters:
  • relative_url (str) – the non-domain part of the URL, for example "/samples/10-TB-1". “Relative” may be misguiding here: only the domain is omitted.
  • data (dict mapping str to str, int, float, bool, file, or list) – the POST data, or None if it’s supposed to be a GET request.
  • response_is_json (bool) – whether the content type of the response must be JSON
Returns:

the response to the request

Return type:

object

Raises:
  • JuliaBaseError – if JuliaBase couldn’t fulfill the request because it contained errors. For example, you requested a sample that doesn’t exist, or the transmitted measurement data was incomplete.
  • urllib.error.URLError – if a lower-level error occured, e.g. the HTTP connection couldn’t be established.
exception JuliaBaseError(error_code, message)

Exception class for high-level JuliaBase errors.

Variables:
  • error_code – The numerical error code. See jb_common.utils for further information, and the root __init__.py file of the various JuliaBase apps for the tables with the error codes.
  • error_message – A description of the error. If error_code is 1, it contains the URL to the error page (without the domain name).
logout()

Logs out of JuliaBase.

class PrimaryKeys

Dictionary-like class for storing primary keys. I use this class only to delay the costly loading of the primary keys until they are really accessed. This way, GET-request-only usage of the Remote Client becomes faster. It is a singleton.

Variables:components – set of types of primary keys that should be fetched. For example, it may contain "external_operators=*" if all external operator’s primary keys should be fetched. The modules the are part of jb_remote are supposed to populate this attribute in top-level module code.
setup_logging(destination=None)

If the user wants to call this in order to enable logging, he must do so before logging in. Note that it is a no-op if called a second time.

Parameters:destination (str) –

Where to log to; possible values are:

"file"
Log to /tmp/jb_remote.log if /tmp is existing, otherwise (i.e. on Windows), log to jb_remote.log in the current directory.
"console"
Log to stderr.
None
Do not log.
parse_timestamp(timestamp)

Convert a timestamp coming from the server to a Python datetime object. The server serialises with the DjangoJSONEncoder, which in turn uses the .isoformat() method. Note that the timestamp must use the UTC timezone (Z for zulu), which seems to be always the case for non-template responses (at least for PostgreSQL and SQLite).

Parameters:timestamp (str) – the timestamp to parse, coming from the server. It must have ISO 8601 format format exlicitly with the “Z” timezone.
Returns:the timestamp as a Python datetime object
Return type:datetime.datetime
double_urlquote(string)

Returns a double-percent-quoted string. This mimics the behaviour of Django, which quotes every URL retrieved by django.urls.resolve. Because it does not quote the slash “/” for obvious reasons, I have to quote sample names, sample series names, deposition numbers, and non-int process “identifying fields” before they are fed into resolve (and quoted again).

format_timestamp(timestamp)

Serializses a timestamp. This is the counter function to parse_timestamp, however, there is an asymmetry: Here, we don’t generate ISO 8601 timestamps (with the "T" inbetween). The reason is that Django’s DateTimeField would not be able to parse it. But see <https://code.djangoproject.com/ticket/11385>.

Parameters:timestamp (datetime.datetime) – the timestamp to format
Returns:the timestamp in the format “YYYY-MM-DD HH:MM:SS”.
Return type:str
sanitize_for_markdown(text)

Convert a raw string to Markdown syntax. This is used when external (legacy) strings are imported. For example, comments found in data files must be sent through this function before being stored in the database.

Parameters:text (str) – the original string
Returns:the Markdown-ready string
Return type:str
as_json(value)

Prints the value in JSON fromat to standard output. This routine comes in handy if Python is called from another program which parses the standard output. Then, you can say

as_json(User("r.calvert").permissions)

and the calling program (written e.g. in Delphi or LabVIEW) just has to convert JSON into its own data structures.

Parameters:value (object (an arbitrary Python object)) – the data to be printed as JSON.
class TemporaryMySamples(sample_ids)

Context manager for adding samples to the “My Samples” list temporarily. This is used when editing or adding processes. In order to be able to link the process with samples, they must be on your “My Samples” list.

This context manager should be used linke this:

with TemporaryMySamples(sample_ids):
    ...

The code at ... can safely assume that the sample_ids have been added to “My Samples”. After having execuded this code, those samples that hadn’t been on “My Samples” already are removed from “My Samples”. This way, the “My Samples” list is unchanged eventually.

param sample_ids:
 the IDs of the samples that must be on the “My Samples” list; it my also be a single ID
class Sample(name=None, id_=None)

Class representing samples.

Parameters:
  • name (str) – the name of an existing sample; it is ignored if id_ is given
  • id (int) – the ID of an existing sample
submit()

Submit the sample to the database.

Returns:the sample’s ID if succeeded.
Return type:int
class Result(id_=None, with_image=True)

Class representing result processes.

Parameters:
  • id (int or str) – if given, the instance represents an existing result process of the database. Note that this triggers an exception if the result ID is not found in the database.
  • with_image (bool) – whether the image data should be loaded, too
submit()

Submit the result to the database.

Returns:the result process ID if succeeded.
Return type:int
class User(username)

Class representing a user.

Parameters:username (str) – the user’s login name
id

Contains the user’s ID as an int.

permissions

Contains the user’s permissions. It is a set of names of the form “app_label.codename”.

topics

Contains the topics the user is a member of. It is a set of topic IDs. In particular, it is not their names, as they are subject to change.