9.6 KiB
20231013:
Adding the shipcall evaluation (traffic state). Provided a one-line function to connect, evaluate and update the shipcalls in a database The sql database can now be properly updated by SQL calls (using mysql.connector and pydapper) Preparing a pull request, where all necessary changes are provided.
TODO: there must be a call function in the application, which asks for an update of the evaluation state & message. Discussing with Daniel Schick.
20231012:
Building out the remaining validation rules, which are all finalized now (0001 - 0006). Made sure that the ValidationRuleState.NONE is 0. Within the notification check, a notification requirement is based on new_state>old_state. To prevent future none->green changes to trigger a notification, a 'max' call is used. When max(GREEN, NONE), then the old state is considered at least 'green'. This solves the issue.
Renaming 'ValidationRuleState' to StatusFlags, which better suits the client version and improves understanding. The traffic status (StatusFlags) will be stored in 'shipcall.evaluation'. The potential error message (str) is stored in 'shipcall.evaluation_message'. Create a oneline function that applies all validation rules and stores the results in 'evaluation' and 'evaluation_message'.
20231006
Focusing entirely on the traffic light. The ValidationRules object is inherited from ValidationRuleFunctions, which is connected to the SQLHandler. This makes it possible to compute all functions within ValidationRules. The object's methods have a default naming convention and input/output convention. The outputs always return a tuple of (enumeration object, method name). The method name is computed dynamically and can be used to indicate, what violation took place. The names are chosen in a way, where they are understandable immediately. However, it might still make sense to map these to genuine error messages, that users can understand better (and use German language). The functions were mapped out and the necessary variables were selected. Finally, a shipcall object is enough to select the respected 'times'. It is still somewhat unclear, how the participant's role will be selected (contacted D. Schick).
Creating tests for the validation rules, so the logic is verified. This will need extensive testing and stub objects.
20231004
Working on the pandas dataframes, obtained from the mysql database. The list of participants is loaded and automatically assigned to each shipcall. A blank column 'validation_state' (str) and 'validation_state_changed' (datetime) are added upon initialization. Created methods and tests for simple retrieval of data models based on the pandas dataframe. This allows for quick and easy connection to any rule-based verification system (e.g., ValidationRules or InputValidation). (Note & #TODO: according to Christin, there should be LocalValidationRules and GlobalValidationRules. Local: focus only on a single shipcall & time, Global: compare multiple shipcalls & times)
Konfliktbehebung nach dem Merge mit 'develop'.
20230929
Removing 'tug' from the library due to feedback of Daniel Schick. Creating an InputValidation object, which uses a dictionary of supported dataclass models. It maps directly towards the respective input validation objects of the dataclass, so the correct rule-set is applicable. Creating RequestStatusCode objects (parent and child classes) for the default status codes within the Flask HTTP framework (200, 201, 400, 403, 404, 500). This ensures that responses always follow the default format. It also makes debugging easier and more reproducible. Creating an SQLHandler object, which can read data from the SQL database and store it in a pandas DataFrame. These dataframes are much easier to handle and provide access to numpy, which allows for simple and quick boolean comparisons or computations.
20230915
Installed the 'schedule' library, which proves access to continuous routines in defined intervals. The library will be particularly useful to monitor alarm event notifications, as the threshold (e.g., 30 minutes) can be taken into account. When creating such a schedule object, an interval of 60 seconds may be sufficient, so that all notifications are checked every minute, whether they are eligible to be released (e.g., via E-Mail).
Created a queue item (collections.deque) to store notifications in the queue. This queue has been connected to the 'schedule', so one can monitor it in fixed intervals. The queue item is likely going to be shifted towards NotificationList instead.
Created a helper function and tests for time differences, so the different between an event's .modified date to the current time can quickly be obtained. A respective notification stub object has been provided, so testing becomes straight forward.
created a suggestion to update the list objects for dataclasses (e.g., BerthList)
apt installed - MySQL Workbench (see brecal/misc/mysql-workbench-community_8.0.34-1ubuntu22.04_amd64.deb) - heidisql-wine - mysql-server - mysql-client https://stackoverflow.com/questions/39281594/error-1698-28000-access-denied-for-user-rootlocalhost scope, Max123 ---- HalloWach_2323XXL!! ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'HalloWach_2323XXL!!';
20230911
Added git SSH key, updated the documentation. Successfully connected to the brecal git repository. The configurations are successful.
20230908
Created a library 'brecal_utils', which will contain some assisting functions and reusable objects. This also enables creating a tests/ folder. The setup.py file currently does not suggest the use of a license ('to be determined'). (#TODO_license: clarify)
init & _version are created and refer to another. The _version suggests '0.0.1' initially.
Creating a conda environment 'brecal' and an environment.yml file. Preparing the environment.yml file to contain only relevant libraries (for now) and create a {brecal} conda environment.
Auto-installing the local lib_brecal_utils package via pip. However, the path to that file and the conda environment are currently hardcoded (#TODO_hardcoded_conda_paths)
Setup the git connection, local configurations and pre-conditioned access to the git remote repository. (#TODO_SSH_KEY: Will talk to Daniel about the connection key) (20230911: solved)
Adding file_handling.py and test_handling.py with updated docstrings to the library. These are copied from other projects and give baseline functionality, that is commonly used in my workflow. Also encorporating the notebook {run_unit_tests.ipynb}, which will, together with the {notebooks_metz} folder be ignored for .git versioning.
Removing the 'jwt' dependency in server/requirements.txt, as it raised an error on my operation system. This is likely caused by jwt and PyJWT overlapping. When removing it from the requirements.txt, the flask-jwt-extended dependency automatically pulls the correct PyJWT version (instead of jwt).
Definition of rules for validation as pre-suggestions in a jupyter notebook. Converting the 'Liegeplätze_sample_format.xlsx' to a .csv file, which may be used in the validation rules (mapping between berth and terminal).
Setting up stub object for every currently defined dataclass. These will serve as the foundation for the traffic-light tests. Created an object {ValidationRules}, which takes shipcalls and times as variables and determines validation_state and notification_state. This will likely be the foundation for the traffic light system.
Created a wrapper for validation function, whereas every function could return a tuple of (validation_state:bool, validation_name:str). This could be used as an indicator to determine the root cause of a traffic light issue.
Suggestion: there is a library, which seems interesting. It builds on marshmallow and extends it by 'validators', which can be applied directly on the marshmallow.field objects. (https://apiflask.com/schema/). This requires the apiflask dependency.
Adding schema_validation functions (Ship and Participant), time_logic (will be used in the validation_rules later), validation_rules (covers time-overlap handling). Further, creating simple unit tests for stubs (will be extended soon). (note: copied this text from the commit message)
Creating a large, very extensive fleet of unit tests for the Ship object and its keys.
################ Continuous Building ################
Ensure that the active environment can run mamba installations (much preferably the 'base' environment) Note: this part of the documentation is incomplete. The commands currently only cover updating an existing environment.
1.) Locate to the project root folder cd ~/brecal 2.) Update the environment. Note: it is explicitly mandatory to use the --prefix argument, which pin-points towards the correct conda environment (in this case {brecal}), as otherwise mamba does not understand the location of the environment and does not update the packages. PIP_NO_DEPS=1 ignores the dependencies of every pip package. Make sure to incorporate these packages in the conda dependencies then, which is much cleaner. PIP_NO_DEPS=1 mamba env update --file environment.yml --prefix /home/scope/anaconda3/envs/brecal
Note: (UNTESTED): to create an environment instead, use this:
PIP_NO_DEPS=1 mamba env create --prefix [conda_path]/envs/[new_env_name] -f [environment_file]
Building Steps
Create the environment, activate it, and install mamba before continuing, as it is much faster. The locate the environment yml and install
conda create --name brecal python=3.11
conda activate brecal
conda install -c conda-forge mamba
cd ~/brecal
PIP_NO_DEPS=1 mamba env update --file environment.yml --prefix /home/scope/anaconda3/envs/brecal