October 21, 2020

I began working on the “Yahoo Finance Stock Ticker App” project.

I’m planning on assigning a “Scrapy” based agent to each of the pages, and to consistently return the same data feed.

Here is a link to the related “Scrapy” Python 3 module:

I’m using this instead of BeautifulSoup 4 as I’d like multiple agents to send asynchronous requests instead of just one at a time.

I’d like maybe deploy this with some kind of lightweight backend program like ‘Flask’ in combination with React to make it sort of a hybrid between Python 3 and JS.

We’ll see how it turns out :)

I’ve learned how to use the ‘venv’ Python 3 command to store a virtual environment for this project as well, as I usually have had the bad habit to just pull in dependencies for cron job projects.

With this approach, you can easily deploy the same project to multiple platforms without having to run a million pip3 commands to make sure that dependencies are pulled in, since ‘requirements.txt’ takes care of that for you.

To get started, make sure you first ‘cd’ into your project:

cd /path/to/your/project

Here’s the command to get started using ‘venv’ to create a virtual environment for your Python 3 project by utilizing the ‘python3 -m’ parameter so that we can focus on utilizing the script from one specific module, which is ‘venv’, aka the first instance of ‘venv’ in the following command. NOTE: The second ‘venv’ parameter can be named whatever you want, though usually, its a convention to name any Python 3 virtual environments as’venv’:

python3 -m venv venv

Here’s the major ‘venv’ command to activate a virtual environment once you have entered your project directory:

source venv/bin/activate

Once you activate your virtual environment (“venv”), then use ‘pip3 install’ and install any required packages. In my case, I did ‘pip3 install scrapy’ for example so that I can work with that module for this project.

Here’s the related command to store all the ‘pip3’ packages into ‘requirements.txt’:

pip3 freeze > requirements.txt

The most important thing is to NOT add your ‘venv’ to your actual .git directory if you plan on using ‘git’ as your version source control method, as you only need ‘requirements.txt’, and the user just needs the ‘requirements.txt’ file to build the ‘venv’ themselves.

Here’s the command to exit the ‘venv’ virtual environment:

deactivate

Here’s another scenario: Let’s say that you wanted to utilize the ‘pip3’ modules that are native and already present on the current machine but ADD to those packages for a given project.

You can instead use the following command, which allows you to add more ‘pip3’ dependencies for that project by initially piggybacking off the already existing global dependencies and allows you to add in new ‘pip3’ dependencies JUST for that given project:

python3 -m venv venv --system-site-packages

If you wanted to list the local dependencies in that above scenario, you can utilize the following command:

pip3 list --local

It’s interesting that I’ve gone through the entire gambit of a lot of Linux based installs, both OS-level, and just installing Linux programs (by compiling from source, etc), and something like Python 3’s ‘venv’ virtual environment just makes sense to me. I never used it before because I thought it was useless, but after having to revive my old Python 3 project websites for probably the third time after a site re-design, I realized that its the best way going forward to deploy Python 3 apps.

On a separate note, I found a pretty cool “public access” UNIX system that allows you to access remote shells, and shared logins. Very cool concept, might play with this sometime this week:

Here’s to learning from your mistakes!

~Sam