Setting Up Your Python Project Environment
When starting a new Python project, setting up a clean and organized environment is crucial for maintaining your code and managing dependencies. In this blog, we’ll walk through the steps to set up your Python project environment using either pip
or conda
, including creating a virtual environment, installing essential libraries, and ensuring reproducibility. We’ll also introduce a basic folder structure, utility classes for managing environment variables and logging, and instructions for setting up version control with Git.
Folder Structure
First, let’s define a basic folder structure for your project. This will help keep your code organized and make it easier to manage as your project grows.
project_root/
│
├── utils/
│ ├── __init__.py
│ ├── path_setup.py
│ ├── env_manager.py
│ └── logging_wrapper.py
│
├── notebooks/
│ └── example_notebook.ipynb
│
├── scripts/
│ └── example_script.py
│
├── data/
│ └── example_data.csv
│
├── models/
│ └── example_model.pkl
│
├── main.py
├── .env
├── environment.yml
├── requirements.txt
├── .gitignore
└── README.md
Getting the Code
You can find the complete code for this project in the GitHub repository linked below:
https://github.com/ghudeihed/python_project_setup
Installation Instructions
To get started with your Python project, follow these steps to set up the necessary environment and install essential packages.
Using pip
- Upgrade pip Ensure you have the latest version of pip installed to avoid compatibility issues:
python -m pip install --upgrade pip
- Create a Virtual Environment Create a virtual environment to manage dependencies separately from your system Python installation:
python -m venv project-env
- Activate the Virtual Environment Activate the virtual environment you just created:
- On Windows:
.\project-env\Scripts\activate
- On macOS and Linux:
source project-env/bin/activate
- Install Required Packages Install basic libraries within the virtual environment:
pip install numpy pandas matplotlib jupyter ipykernel python-dotenv
- Generate
requirements.txt
Save the installed packages to arequirements.txt
file:
pip freeze > requirements.txt
- Install Dependencies Using
requirements.txt
If setting up on another machine, install the dependencies fromrequirements.txt
:
pip install -r requirements.txt
Using conda
- Create a New Conda Environment Create a new conda environment with Python 3.10:
conda create -n project-env python=3.10 -y
- Activate the Conda Environment Activate the newly created environment:
conda activate project-env
- Install Required Packages Install basic libraries using conda-forge:
conda install -c conda-forge numpy pandas matplotlib jupyter ipykernel python-dotenv -y
- Generate
environment.yml
Export the environment to anenvironment.yml
file:
conda env export -n project-env > environment.yml
- Install Environment Using
environment.yml
If setting up on another machine, create the environment fromenvironment.yml
:
conda env create -f environment.yml
Setting Up Git
- Initialize Git Repository Initialize a new Git repository in your project directory:
git init
- Create a
.gitignore
File Create a.gitignore
file to exclude files and directories that you don’t want to track in version control
- Add and Commit Files Add and commit your files to the Git repository:
git add .
git commit -m "Initial commit"
Utility Classes
utils/path_setup.py
This utility class helps manage the system path, ensuring that the project root is included in the sys.path
.
# utils/path_setup.py
import os
import sys
class PathSetup:
@staticmethod
def add_project_root_to_sys_path():
project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if project_root not in sys.path:
sys.path.insert(0, project_root) # Insert at the beginning of sys.path
utils/env_manager.py
This utility class helps manage environment variables, making it easier to handle configurations.
# utils/env_manager.py
import os
from dotenv import load_dotenv
from utils.logging_wrapper import LoggingWrapper
logger = LoggingWrapper(__name__).logger
class EnvManager:
"""
A singleton class to manage environment variables and the PYTHONPATH setup.
This class ensures that environment variables are loaded from a .env file
and the project root is added to the PYTHONPATH. It uses the singleton pattern
to ensure the environment is loaded only once.
"""
_instance = None
_env_loaded = False
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(EnvManager, cls).__new__(cls, *args, **kwargs)
return cls._instance
def __init__(self, file_path: str = '.env'):
if not self._env_loaded:
self.load_env(file_path)
def load_env(self, file_path: str) -> None:
if not EnvManager._env_loaded:
try:
if load_dotenv(file_path):
logger.info(f"Environment variables loaded from {file_path}")
else:
logger.warning(f"No environment variables found in {file_path}")
EnvManager._env_loaded = True
except Exception as e:
logger.error(f"Error loading environment variables from {file_path}: {e}")
raise RuntimeError(f"Failed to load environment variables from {file_path}") from e
@staticmethod
def get_env_variable(name: str, default: str = None) -> str:
try:
value = os.getenv(name, default)
if value is None:
logger.warning(f"Environment variable {name} not found, using default value: {default}")
return value
except Exception as e:
logger.error(f"Error getting environment variable {name}: {e}")
raise RuntimeError(f"Failed to get environment variable {name}") from e
utils/logging_wrapper.py
This utility class provides a simple wrapper around Python’s logging module, allowing for consistent and configurable logging throughout your project.
# utils/logging_wrapper.py
import logging
import os
from datetime import datetime
from logging.handlers import RotatingFileHandler
class LoggingWrapper:
"""
A wrapper for the Python logging module that sets up logging to both a file and the console.
Attributes:
logger (logging.Logger): The logger instance.
"""
def __init__(self, name, log_dir='logs
', log_level=logging.DEBUG, console_level=logging.ERROR, max_bytes=5*1024*1024, backup_count=5):
self.logger = logging.getLogger(name)
self.logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
if not os.path.exists(log_dir):
os.makedirs(log_dir)
now = datetime.now().strftime('%Y%m%d_%H%M%S')
log_file = os.path.join(log_dir, f'log_{now}.log')
fh = RotatingFileHandler(log_file, maxBytes=max_bytes, backupCount=backup_count)
fh.setLevel(log_level)
fh.setFormatter(formatter)
ch = logging.StreamHandler()
ch.setLevel(console_level)
ch.setFormatter(formatter)
if not self.logger.handlers:
self.logger.addHandler(fh)
self.logger.addHandler(ch)
def debug(self, msg, *args, **kwargs):
self.logger.debug(msg, *args, **kwargs)
def info(self, msg, *args, **kwargs):
self.logger.info(msg, *args, **kwargs)
def warning(self, msg, *args, **kwargs):
self.logger.warning(msg, *args, **kwargs)
def error(self, msg, *args, **kwargs):
self.logger.error(msg, *args, **kwargs)
def critical(self, msg, *args, **kwargs):
self.logger.critical(msg, *args, **kwargs)
Example Script
Here’s a toy example for example_script.py
that utilizes the utility classes (PathSetup
, EnvManager
, and LoggingWrapper
) to demonstrate their benefits. The script will read an environment variable, perform a simple calculation, log the results, and include error handling.
# scripts/example_script.py
from utils import EnvManager, LoggingWrapper
# Initialize logging
logger = LoggingWrapper(__name__).logger
# Initialize environment manager and load .env variables
env_manager = EnvManager()
def main():
try:
# Get environment variables
number1 = env_manager.get_env_variable('NUMBER1', '10')
number2 = env_manager.get_env_variable('NUMBER2', '5')
# Convert to integers
num1 = int(number1)
num2 = int(number2)
logger.info(f"Retrieved environment variables: NUMBER1={num1}, NUMBER2={num2}")
# Perform a simple calculation
result = num1 + num2
logger.info(f"The result of adding {num1} and {num2} is {result}")
except ValueError as e:
logger.error(f"Error converting environment variables to integers: {e}")
except Exception as e:
logger.error(f"An unexpected error occurred: {e}")
if __name__ == "__main__":
main()
Entry Point Script
Here’s the entry point script main.py
that sets up the environment and calls the example script.
# main.py
from utils.path_setup import PathSetup
PathSetup.add_project_root_to_sys_path()
from scripts.example_script import main
if __name__ == "__main__":
main()
Sample .env
File
Make sure your .env
file contains the following entries for the example to work:
# .env
NUMBER1=15
NUMBER2=25
Code language: PHP (php)
Dependencies List
numpy
pandas
matplotlib
jupyter
ipykernel
python-dotenv
Additional Steps
- Verify Installation Ensure all packages are installed correctly by starting a Jupyter notebook:
jupyter notebook
- Updating the Environment To update the environment with new packages, install them and regenerate the
requirements.txt
orenvironment.yml
file as appropriate. For pip:
pip install <new-package>
pip freeze > requirements.txt
For conda:
conda install -c conda-forge <new-package> -y
conda env export -n project-env > environment.yml
Conclusion
Setting up a well-organized Python project environment is essential for efficient development and collaboration. By following these steps and using the provided utility classes and folder structure, you can ensure your project remains manageable and reproducible. Whether you prefer pip
or conda
, these instructions will help you get started quickly and maintain a clean environment for your Python projects.