Loosen deps

README.md

@@ -90,9 +90,7 @@ configure model, and connect to it.
     * [Utility to generate OpenAPI schema](#utility-to-generate-openapi-schema)
         * [Path](#path)
         * [Usage](#usage-1)
-    * [Utility to generate `requirements.*` files](#utility-to-generate-requirements-files)
-        * [Path](#path-1)
-        * [Usage](#usage-2)
+    * [Generating `requirements.txt` files](#generating-requirements.txt-files)
         * [Known issue](#known-issue)
     * [Uploading artifact containing the pytest results and configuration to an s3 bucket.](#uploading-artifact-containing-the-pytest-results-and-configuration-to-an-s3-bucket)
         * [Path](#path-2)
@@ -891,55 +889,34 @@ This script re-generated OpenAPI schema for the Lightspeed Service REST API.
 ### Usage
 
 ```
-pdm generate-schema`
+pdm generate-schema
 ```
 
-## Utility to generate `requirements.*` files
+## Generating requirements.txt files
 
-Generate list of packages to be prefetched in Cachi2 and used in Konflux for hermetic build.
+For Konflux hermetic builds, Cachi2 uses the `requirements.txt` format to generate a list of packages to prefetch.  
 
-This script performs several steps:
+To generate the `requirements.txt` file, follow these steps:  
 
-1. removes torch+cpu dependency from project file
-2. generates requirements.txt file from pyproject.toml + pdm.lock
-3. removes all torch dependencies (including CUDA/Nvidia packages)
-4. downloads torch+cpu wheel
-5. computes hashes for this wheel
-6. adds the URL to wheel + hash to resulting requirements.txt file
-7. downloads script `pip_find_builddeps` from the Cachito project
-8. generated requirements-build.in file
-9. compiles requirements-build.in file into requirements-build.txt file
+1. Run `pdm update` – updates dependencies to their latest versions allowed by our `pyproject.toml` pins.  
+2. Run `pdm lock` – generates the `pdm.lock` file used to sync development and CI (prow) environments.  
+3. Run `pdm lock --platform=manylinux_2_34_x86_64 --lockfile=pdm-manylinux.lock` – generates the `pdm-manylinux.lock` file with hashes only for the specified architecture (reduces image size).  
+4. Run `pdm export -f requirements -o requirements.txt --lockfile=pdm-manylinux.lock` – exports the architecture-specific lock file into `requirements.txt`.  
 
-Please note that this script depends on tool that is downloaded from repository containing
-Cachito system. This tool is run locally w/o any additional security checks etc. so some
-care is needed (run this script from within containerized environment etc.).
+> There might be more than one architecture-specific lock and requirements files.
 
-### Path
-
-[scripts/generate_packages_to_prefetch.py](scripts/generate_packages_to_prefetch.py)
-
-### Usage
+### Known Issue  
 
-```
-usage: generate_packages_to_prefetch.py [-h] [-p]
-
-options:
-  -h, --help            show this help message and exit
-  -p, --process-special-packages
-                        Enable or disable processing special packages like torch etc.
-  -c, --cleanup         Enable or disable work directory cleanup
-  -w WORK_DIRECTORY, --work-directory WORK_DIRECTORY
-                        Work directory to store files generated during different stages
-                        of processing
-```
+The `sqlalchemy` and `httpx` packages are being removed from `requirements.txt` in favor of "extras." However, this doesn't work well with Konflux, so when generating `requirements.txt`, you need to manually revert these changes.  
 
-### Known issue
+#### Updating `sqlalchemy` and `httpx`  
 
-When SQLAlchemy package is not locked to latest version in `pyproject.toml` and `pdm.lock`, this script will fail due to issue in `pip`. To fix this issue it is needed to follow those steps:
+To update these dependencies:  
 
-1. Look at https://pypi.org/project/SQLAlchemy/ to retrieve latest SQLAlchemy version
-1. Update `pyproject.toml` file accordingly using `SQLAlchemy=={latest_version}`
-1. Run `pdm update sqlalchemy`
+- Unlock them in `pyproject.toml` and run `pdm update httpx sqlalchemy` to get the latest possible versions.  
+- Check the lock file and pin the versions again in `pyproject.toml`.  
+- After generating `requirements.txt`, manually revert the changes where these packages are removed in favor of extras.  
+- Update the hashes according to what is in the lock file (some manual comparison of hashes is needed).  
 
 
 ## Uploading artifact containing the pytest results and configuration to an s3 bucket.