Where are Superset logs stored?

Superset logs, essential for monitoring and troubleshooting, are primarily stored in a location that varies based on the deployment environment and specific configurations.

Default Superset Log Location

By default, the main Superset application log file, where the application records its operational details, warnings, and errors, is typically located in the ~/superset directory within the user's home path. This default log file often goes by a name like superset.log.

Configuring Superset Log Storage

Superset's logging behavior is highly customizable through its superset_config.py file, which allows administrators to define log file paths, logging levels, and handlers. This configuration flexibility means that while a default location exists, logs are often redirected to other directories or logging systems, especially in production environments.

Key configuration parameters related to logging often include:

LOG_FILE: Specifies the absolute path for the main Superset log file.
LOGGING_CONFIG: Allows for a full Python dictionary-based logging configuration, providing granular control over loggers, handlers, and formatters.
SUPERSET_HOME: Although not directly for logs, this environment variable can influence the base directory for default Superset files, potentially affecting log placement if relative paths are used.

Example superset_config.py snippet for custom log path:

# superset_config.py
# Example: Redirect Superset logs to a specific directory
LOG_FILE = '/var/log/superset/superset_app.log'

# Example: Basic logging configuration
import logging
from logging.handlers import RotatingFileHandler

# Configure root logger
logging.basicConfig(level=logging.INFO)

# Create a rotating file handler
file_handler = RotatingFileHandler(
    LOG_FILE, maxBytes=1024 * 1024 * 100, backupCount=10
) # 100 MB per file, 10 backup files
file_handler.setFormatter(logging.Formatter(
    '%(asctime)s:%(levelname)s:%(name)s:%(message)s'
))

# Add the file handler to the root logger
logging.getLogger().addHandler(file_handler)

# Optional: Set a higher level for specific Superset modules
logging.getLogger('superset').setLevel(logging.DEBUG)

Common Log Types in Superset Deployments

Beyond the main application logs, a typical Superset deployment often generates several other types of logs crucial for a complete operational overview:

Log Type	Description	Typical Location/Source
Application Logs	Core Superset operations, errors, warnings, and informational messages.	`~/superset/superset.log` (default), configurable path in `superset_config.py`, or standard output/error in containers.
Web Server Access Logs	Records of HTTP requests served by the web server (e.g., Gunicorn, Nginx, Apache).	Web server's default log directories (e.g., `/var/log/nginx/access.log`, `/var/log/apache2/access.log`), or container stdout.
Web Server Error Logs	Errors encountered by the web server during request processing.	Web server's default log directories (e.g., `/var/log/nginx/error.log`, `/var/log/apache2/error.log`), or container stderr.
Database Logs	Logs from the metadata database (e.g., PostgreSQL, MySQL) that Superset uses.	Managed by the database server's configuration (e.g., `/var/log/postgresql`, MySQL data directory).

Log Locations in Containerized Environments

In modern deployments using Docker or Kubernetes, the approach to log storage is significantly different, often leveraging container orchestration features.

Docker Containers: By default, applications running inside Docker containers send their logs to stdout (standard output) and stderr (standard error). These logs can be viewed using the docker logs <container_id_or_name> command. For persistent storage or centralized logging, Docker volumes are often mounted to specific directories within the container where the application writes its logs.
Kubernetes Pods: Similar to Docker, applications within Kubernetes pods typically output logs to stdout and stderr. Kubernetes collects these logs, which can then be accessed using kubectl logs <pod_name>. For robust logging in Kubernetes, a common practice is to use a logging agent (like Fluentd, Filebeat, or Logstash) running as a sidecar container or daemonset to collect logs and forward them to a centralized logging system (e.g., Elasticsearch, Loki, Splunk).

Practical Tips for Locating Superset Logs

To pinpoint Superset log files, consider these steps:

Check superset_config.py: Always inspect your superset_config.py file first, as it explicitly defines custom log paths via LOG_FILE or more complex LOGGING_CONFIG settings.
Inspect Container Logs: If Superset is running in Docker or Kubernetes, use docker logs or kubectl logs commands to view real-time output.
Explore Mounted Volumes: In containerized setups, use docker inspect <container_id> or kubectl describe pod <pod_name> to identify any mounted volumes where logs might be persistently stored.
Search Default Locations: Look in the ~/superset directory in the user's home directory.
Examine Web Server Logs: If Superset is served behind a reverse proxy like Nginx or Apache, check their respective log directories (commonly /var/log/nginx/ or /var/log/apache2/) for access and error logs.
Review System Logs: In some cases, application errors not caught by Superset's internal logging might appear in system-level logs (e.g., journalctl on Linux systems).