The remote client¶
The “remote client” is a programming library for contacting JuliaBase over the network. Being a library, it is not a program per se but used by your own code for the bidirectional communication with the database. The remote client communicates through a REST HTTP interface implemented in the JuliaBase server. This interface could be used directly, however, the remote client makes it much easier.
In the following, we will describe what the remote client can do and how to get
it running. For the API itself, we have to refer to the documentation in the
sources of the Python files in remote_client/
. Note especially the
examples in remote_client/examples/
.
Use cases¶
The remote client has many applications.
Crawlers¶
Crawlers are programs that go through a bunch of data or log files and transmit their content to the database. For example, a measurement setup may write a data file for every measurement run. Every hour, a crawler is called, which scans for new files. It reads the new files and uses the content to add new measurements processes to the database. This way, new measurements automatically end up in the database. You may have more than one crawler regularly running in your institute.
The very same crawler can also be used to import legacy data. The crawler can be programmed in such a way that during its first run, it imports thousands of data sets of recent years, and from there on, it imports newly added data sets every hour.
Since crawlers run unattendedly, they must be very robust. If a data file is invalid, or if the central server is temporarily unavailable, it must gently exit and try again next time, without forgetting to add something, and without adding things twice.
Moreover, legacy data mostly is poorly structured and organized. If you think your data is the exception, think again. During writing the crawler, you will be surprised how many odd ideas and mistakes your colleagues are capable of.
Be that as it may, crawlers are the primary use case for the remote client. The remote client contains functions especially for making writing them as easy as possible.
Connecting the setup with the database¶
A more direct way than the crawler for bringing processing and measurement data into the database, albeit not always feasible, is to extend the control program of the experimental setup. For this, you must have access to the source code of the control program. Then, you can use the remote client for writing new runs immediately into the database.
However, the communication needn’t be one-way. You can also use the remote client to authenticate the user, to check whether he or she is allowed to use this setup, and to check whether the sample names belong to valid and for the user accessible samples.
The remote client is written in Python, however, it is easy to use it from programs written in other programming languages, and JuliaBase ships with bindings to Delphi, LabVIEW, and Visual Basic.
Data mining and analysis¶
Everyone with browser access to the JuliaBase database can use the remote client for accessing, too. The permissions will be the same in both cases, of course. This can be used by researchers capable of programming to get data from the database and do something with it, e.g. creating statistics or evaluating raw data.
Imagine the following: A researcher makes many sample series and measures their temperature-dependent conductivity. The researcher can write a program that extracts the names of the raw data files from the database, then creates a plot for each sample series with the conductivity curves of all samples of that series, and writes these plots back to the database. There, one can see them on the series’ pages, and in every sample’s data sheet.
Extending the remote client¶
As for JuliaBase itself, also the remote client should be extended with the functionality special to your institution. This is not absolutely necessary, but without it, you can only use the very basic functionality. However, extending the remote client is much easier than extending JuliaBase.
In JuliaBases’ source code, the default remote client resides in the directory
remote_client/
, and it has the following structure:
remote_client/
jb_remote_inm.py
jb_remote/
__init__.py
common.py
samples.py
settings.py
...
Replace the file jb_remote_inm.py
with your own institute’s code,
and give your file a name derived from your institution’s name. Your file
should start with something like:
from jb_remote import *
settings.ROOT_URL = settings.TESTSERVER_ROOT_URL = "https://juliabase.my-institute.edu/"
From there on, you are totally free how to program your incarnation of the
remote client. You may use jb_remote_inm.py
as a source of
inspiration, of course.
Settings¶
The following settings in the remote client are available and should be set in
your jb_remote_inm.py
:
ROOT_URL
The URL of the production server. It must end in a slash. Default:
None
TESTSERVER_ROOT_URL
The URL of the test server. It must end in a slash. Default:
"https://demo.juliabase.org/"
SMTP_SERVER
The DNS name of the SMTP server used for outgoing mail. It may be used in crawlers to send success or error emails. You may add a port number after a colon. Default:
"mailrelay.example.com:587"
SMTP_LOGIN
The login name of the SMTP server. If empty, no login is performed. If not empty, TLS is used. Default:
"username"
SMTP_PASSWORD
The password used to login to the SMTP server through TLS. Default:
"password"
EMAIL_FROM
The sender used for outgoing mails. Default:
"me@example.com"
EMAIL_TO
The recipient of outgoing mails. Default:
"admins@example.com"
Local installation and usage¶
Once you finished extending the default remote client, your remote client
consists of the files listed above, with
jb_remote_inm.py
substituted by your file called, say,
jb_remote_my_institute.py
. This set of files needs to be copied to the machines
where the remote client is supposed to be used, and the top directory (the one
with your jb_remote_my_institute.py
) should be in PYTHONPATH
.
Then, you use the remote client with a Python script as easy as this:
from jb_remote_my_institute import *
setup_logging("console")
login("juliabase", "12345")
sample = Sample("14-JS-1")
sample.current_location = "Sean's office"
sample.edit_description = "location changed"
sample.submit()
logout()
By the way, the files are organized in a way that you can update very
conveniently: If a new version of JuliaBase is released, you simply have to
replace the jb_remote/
subdirectory with the new one.
Classes and functions¶
As explained in Extending the remote client, the following names become available by saying
from jb_remote import *
- connection¶
The connection to the database server. It is of the type
JuliaBaseConnection
.
- primary_keys¶
A dict-like object of type
PrimaryKeys
that is a mapping of identifying keys to IDs. Possible keys are:"users"
mapping user names to user IDs.
"external_operators"
mapping external operator names to external operator IDs.
"topics"
mapping topic names to topic IDs.
- class JuliaBaseConnection¶
Class for the routines that connect to the database at HTTP level. This is a singleton class, and its only instance resides at top-level in this module.
- open(relative_url, data=None, response_is_json=True)¶
Do an HTTP request with the JuliaBase server. If
data
is notNone
, its a POST request, and GET otherwise.- Parameters:
relative_url (str) – the non-domain part of the URL, for example
"/samples/10-TB-1"
. “Relative” may be misguiding here: only the domain is omitted.data (dict mapping str to str, int, float, bool, file, or list) – the POST data, or
None
if it’s supposed to be a GET request.response_is_json (bool) – whether the content type of the response must be JSON
- Returns:
the response to the request
- Return type:
object
- Raises:
JuliaBaseError – if JuliaBase couldn’t fulfill the request because it contained errors. For example, you requested a sample that doesn’t exist, or the transmitted measurement data was incomplete.
urllib.error.URLError – if a lower-level error occured, e.g. the HTTP connection couldn’t be established.
- exception JuliaBaseError(error_code, message)¶
Exception class for high-level JuliaBase errors.
- Variables:
error_code – The numerical error code. See
jb_common.utils
for further information, and the root__init__.py
file of the various JuliaBase apps for the tables with the error codes.error_message – A description of the error. If
error_code
is1
, it contains the URL to the error page (without the domain name).
- class PrimaryKeys¶
Dictionary-like class for storing primary keys. I use this class only to delay the costly loading of the primary keys until they are really accessed. This way, GET-request-only usage of the Remote Client becomes faster. It is a singleton.
- Variables:
components – set of types of primary keys that should be fetched. For example, it may contain
"external_operators=*"
if all external operator’s primary keys should be fetched. The modules the are part of jb_remote are supposed to populate this attribute in top-level module code.
- as_json(value)¶
Prints the value in JSON fromat to standard output. This routine comes in handy if Python is called from another program which parses the standard output. Then, you can say
as_json(User("r.calvert").permissions)
and the calling program (written e.g. in Delphi or LabVIEW) just has to convert JSON into its own data structures.
- Parameters:
value (
object
(an arbitrary Python object)) – the data to be printed as JSON.
- double_urlquote(string)¶
Returns a double-percent-quoted string. This mimics the behaviour of Django, which quotes every URL retrieved by
django.urls.resolve
. Because it does not quote the slash “/” for obvious reasons, I have to quote sample names, sample series names, deposition numbers, and non-int process “identifying fields” before they are fed intoresolve
(and quoted again).
- format_timestamp(timestamp)¶
Serializses a timestamp. This is the counter function to
parse_timestamp
, however, there is an asymmetry: Here, we don’t generate ISO 8601 timestamps (with the"T"
inbetween). The reason is that Django’sDateTimeField
would not be able to parse it. But see <https://code.djangoproject.com/ticket/11385>.- Parameters:
timestamp (datetime.datetime) – the timestamp to format
- Returns:
the timestamp in the format “YYYY-MM-DD HH:MM:SS”.
- Return type:
str
- login(username, password, testserver=False)¶
Logins to JuliaBase.
- Parameters:
username (str) – the username used to log in
password (str) – the user’s password
testserver (bool) – whether the testserver should be user. If
False
, the production server is used.
- logout()¶
Logs out of JuliaBase.
- parse_timestamp(timestamp)¶
Convert a timestamp coming from the server to a Python
datetime
object. The server serialises with theDjangoJSONEncoder
, which in turn uses the.isoformat()
method. Note that the timestamp must use the UTC timezone (Z for zulu), which seems to be always the case for non-template responses (at least for PostgreSQL and SQLite).- Parameters:
timestamp (str) – the timestamp to parse, coming from the server. It must have ISO 8601 format format exlicitly with the “Z” timezone.
- Returns:
the timestamp as a Python datetime object
- Return type:
datetime.datetime
- sanitize_for_markdown(text)¶
Convert a raw string to Markdown syntax. This is used when external (legacy) strings are imported. For example, comments found in data files must be sent through this function before being stored in the database.
- Parameters:
text (str) – the original string
- Returns:
the Markdown-ready string
- Return type:
str
- setup_logging(destination=None, filepath=None)¶
Sets up the root logger. Note that it replaces the old root logger configuration fully. Client code should call this function as early as possible.
- Parameters:
destination (str) –
Where to log to; possible values are:
"file"
Log to
/var/lib/crawlers/jb_remote.log
if/var/lib/crawlers
is existing, otherwise (i.e. on Windows), log tojb_remote.log
in the current directory. The directory is configurable by the environment variableCRAWLERS_DATA_DIR
. See also thefilepath
parameter.Logging is appended to that file.
It additionally enables logging to stderr, in order to be useful in containers.
"console"
Log to stderr.
None
Do not log.
filepath (str) – Makes sense only if
destination
is “file”. If given, the log output is sent to this path, and to stderr.
- class Result(id_=None, with_image=True)¶
Class representing result processes.
- Parameters:
id (int or str) – if given, the instance represents an existing result process of the database. Note that this triggers an exception if the result ID is not found in the database.
with_image (bool) – whether the image data should be loaded, too
- submit()¶
Submit the result to the database.
- Returns:
the result process ID if succeeded.
- Return type:
int
- class Sample(name=None, id_=None)¶
Class representing samples.
- Parameters:
name (str) – the name of an existing sample; it is ignored if
id_
is givenid (int) – the ID of an existing sample
- submit()¶
Submit the sample to the database.
- Returns:
the sample’s ID if succeeded.
- Return type:
int
- class TemporaryMySamples(sample_ids)¶
Context manager for adding samples to the “My Samples” list temporarily. This is used when editing or adding processes. In order to be able to link the process with samples, they must be on your “My Samples” list.
This context manager should be used like this:
with TemporaryMySamples(sample_ids): ...
The code at
...
can safely assume that thesample_ids
have been added to “My Samples”. After having executed this code, those samples that hadn’t been on “My Samples” already are removed from “My Samples”. This way, the “My Samples” list is unchanged eventually.- param sample_ids:
the IDs of the samples that must be on the “My Samples” list; it my also be a single ID
- class User(username)¶
Class representing a user.
- Parameters:
username (str) – the user’s login name
- property id¶
Contains the user’s ID as an
int
.
- property permissions¶
Contains the user’s permissions. It is a set of names of the form “app_label.codename”.
- property topics¶
Contains the topics the user is a member of. It is a set of topic IDs. In particular, it is not their names, as they are subject to change.