diff --git a/docs/plans/2025-10-26-x-space-downloader-implementation.md b/docs/plans/2025-10-26-x-space-downloader-implementation.md deleted file mode 100644 index 96de23d..0000000 --- a/docs/plans/2025-10-26-x-space-downloader-implementation.md +++ /dev/null @@ -1,1017 +0,0 @@ -# X Space Downloader Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Build a Python CLI tool that downloads X (Twitter) Space recordings and outputs MP3 files. - -**Architecture:** Thin wrapper around yt-dlp with URL validation, error handling, and Click-based CLI. Cookie-based authentication for X access. - -**Tech Stack:** Python, yt-dlp, Click, pytest - ---- - -## Task 1: Project Setup and Dependencies - -**Files:** -- Create: `requirements.txt` -- Create: `setup.py` -- Create: `src/__init__.py` -- Create: `.gitignore` - -**Step 1: Create requirements.txt** - -Create the file with dependencies: - -```txt -yt-dlp>=2024.0.0 -click>=8.0.0 -``` - -**Step 2: Create setup.py** - -Create the file to make the package installable: - -```python -from setuptools import setup, find_packages - -setup( - name="x-space-downloader", - version="0.1.0", - packages=find_packages(), - install_requires=[ - "yt-dlp>=2024.0.0", - "click>=8.0.0", - ], - entry_points={ - "console_scripts": [ - "x-space-dl=src.main:cli", - ], - }, - python_requires=">=3.8", -) -``` - -**Step 3: Create src package init file** - -```bash -mkdir -p src -touch src/__init__.py -``` - -**Step 4: Create .gitignore** - -```txt -# Python -__pycache__/ -*.py[cod] -*$py.class -*.so -.Python -env/ -venv/ -ENV/ -*.egg-info/ -dist/ -build/ - -# IDEs -.vscode/ -.idea/ -*.swp -*.swo - -# Testing -.pytest_cache/ -.coverage -htmlcov/ - -# Downloads -*.mp3 -*.m4a -cookies.txt -``` - -**Step 5: Install in development mode** - -Run: `pip install -e .` -Expected: Package installed successfully - -**Step 6: Commit** - -```bash -git add requirements.txt setup.py src/__init__.py .gitignore -git commit -m "feat: add project setup and dependencies" -``` - ---- - -## Task 2: URL Validator (TDD) - -**Files:** -- Create: `tests/test_validator.py` -- Create: `src/validator.py` - -**Step 1: Create test directory** - -```bash -mkdir -p tests -touch tests/__init__.py -``` - -**Step 2: Write failing test for valid URL** - -Create `tests/test_validator.py`: - -```python -import pytest -from src.validator import validate_space_url - - -def test_validate_space_url_valid(): - """Test that valid X Space URL is accepted.""" - url = "https://x.com/i/spaces/1ZkKzZWnVRRKv" - is_valid, space_id, error = validate_space_url(url) - - assert is_valid is True - assert space_id == "1ZkKzZWnVRRKv" - assert error is None -``` - -**Step 3: Run test to verify it fails** - -Run: `pytest tests/test_validator.py::test_validate_space_url_valid -v` -Expected: FAIL with "cannot import name 'validate_space_url'" - -**Step 4: Write minimal validator implementation** - -Create `src/validator.py`: - -```python -import re -from typing import Tuple, Optional - - -def validate_space_url(url: str) -> Tuple[bool, Optional[str], Optional[str]]: - """ - Validates X Space URL format. - - Args: - url: The URL to validate - - Returns: - Tuple of (is_valid, space_id, error_message) - """ - # Pattern for X Space URLs - pattern = r"https://x\.com/i/spaces/([a-zA-Z0-9]+)" - match = re.match(pattern, url) - - if not match: - return False, None, "Invalid URL format. Expected: https://x.com/i/spaces/{SPACE_ID}" - - space_id = match.group(1) - return True, space_id, None -``` - -**Step 5: Run test to verify it passes** - -Run: `pytest tests/test_validator.py::test_validate_space_url_valid -v` -Expected: PASS - -**Step 6: Commit** - -```bash -git add tests/__init__.py tests/test_validator.py src/validator.py -git commit -m "feat: add URL validator for X Space URLs" -``` - ---- - -## Task 3: Validator Error Cases (TDD) - -**Files:** -- Modify: `tests/test_validator.py` - -**Step 1: Write test for invalid URL formats** - -Add to `tests/test_validator.py`: - -```python -def test_validate_space_url_invalid_format(): - """Test that invalid URLs are rejected.""" - invalid_urls = [ - "https://twitter.com/i/spaces/1ZkKzZWnVRRKv", # Old twitter.com - "https://x.com/spaces/1ZkKzZWnVRRKv", # Missing /i/ - "https://x.com/i/spaces/", # Missing space ID - "not a url", # Completely invalid - "", # Empty string - ] - - for url in invalid_urls: - is_valid, space_id, error = validate_space_url(url) - assert is_valid is False - assert space_id is None - assert "Invalid URL format" in error -``` - -**Step 2: Run test to verify it passes** - -Run: `pytest tests/test_validator.py::test_validate_space_url_invalid_format -v` -Expected: PASS (implementation already handles these) - -**Step 3: Write test for twitter.com URL support** - -Add to `tests/test_validator.py`: - -```python -def test_validate_space_url_twitter_domain(): - """Test that twitter.com URLs are also accepted.""" - url = "https://twitter.com/i/spaces/1ZkKzZWnVRRKv" - is_valid, space_id, error = validate_space_url(url) - - assert is_valid is True - assert space_id == "1ZkKzZWnVRRKv" - assert error is None -``` - -**Step 4: Run test to verify it fails** - -Run: `pytest tests/test_validator.py::test_validate_space_url_twitter_domain -v` -Expected: FAIL - -**Step 5: Update validator to support twitter.com** - -Modify `src/validator.py`: - -```python -def validate_space_url(url: str) -> Tuple[bool, Optional[str], Optional[str]]: - """ - Validates X Space URL format. - - Args: - url: The URL to validate - - Returns: - Tuple of (is_valid, space_id, error_message) - """ - # Pattern for X Space URLs (supports both x.com and twitter.com) - pattern = r"https://(x|twitter)\.com/i/spaces/([a-zA-Z0-9]+)" - match = re.match(pattern, url) - - if not match: - return False, None, "Invalid URL format. Expected: https://x.com/i/spaces/{SPACE_ID}" - - space_id = match.group(2) # Changed from group(1) to group(2) - return True, space_id, None -``` - -**Step 6: Run all validator tests** - -Run: `pytest tests/test_validator.py -v` -Expected: All tests PASS - -**Step 7: Commit** - -```bash -git add tests/test_validator.py src/validator.py -git commit -m "feat: add validator error cases and twitter.com support" -``` - ---- - -## Task 4: Downloader Module (TDD) - -**Files:** -- Create: `tests/test_downloader.py` -- Create: `src/downloader.py` - -**Step 1: Write failing test for downloader** - -Create `tests/test_downloader.py`: - -```python -import pytest -from unittest.mock import Mock, patch -from src.downloader import download_space - - -def test_download_space_basic(tmp_path): - """Test basic download functionality.""" - space_url = "https://x.com/i/spaces/1ZkKzZWnVRRKv" - - with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class: - # Mock the YoutubeDL instance - mock_ytdl = Mock() - mock_ytdl_class.return_value.__enter__.return_value = mock_ytdl - mock_ytdl.extract_info.return_value = {'id': '1ZkKzZWnVRRKv'} - - # Call download_space - result = download_space( - space_url=space_url, - output_dir=str(tmp_path), - cookies_file=None - ) - - # Verify YoutubeDL was called - mock_ytdl.extract_info.assert_called_once_with(space_url, download=True) - - # Verify result contains expected filename - assert "space_1ZkKzZWnVRRKv" in result -``` - -**Step 2: Run test to verify it fails** - -Run: `pytest tests/test_downloader.py::test_download_space_basic -v` -Expected: FAIL with "cannot import name 'download_space'" - -**Step 3: Write minimal downloader implementation** - -Create `src/downloader.py`: - -```python -import os -import yt_dlp -from typing import Optional - - -def download_space( - space_url: str, - output_dir: str, - cookies_file: Optional[str] = None -) -> str: - """ - Downloads X Space recording using yt-dlp. - - Args: - space_url: URL of the X Space recording - output_dir: Directory to save the MP3 file - cookies_file: Optional path to cookies file for authentication - - Returns: - Path to the downloaded MP3 file - """ - # Extract space ID from URL for filename - space_id = space_url.split('/')[-1] - - # Configure yt-dlp options - ydl_opts = { - 'format': 'bestaudio/best', - 'outtmpl': os.path.join(output_dir, f'space_{space_id}.%(ext)s'), - 'postprocessors': [{ - 'key': 'FFmpegExtractAudio', - 'preferredcodec': 'mp3', - }], - 'quiet': False, - 'no_warnings': False, - } - - # Add cookies if provided - if cookies_file: - ydl_opts['cookiefile'] = cookies_file - - # Download with yt-dlp - with yt_dlp.YoutubeDL(ydl_opts) as ydl: - ydl.extract_info(space_url, download=True) - - # Return the expected output path - return os.path.join(output_dir, f'space_{space_id}.mp3') -``` - -**Step 4: Run test to verify it passes** - -Run: `pytest tests/test_downloader.py::test_download_space_basic -v` -Expected: PASS - -**Step 5: Commit** - -```bash -git add tests/test_downloader.py src/downloader.py -git commit -m "feat: add downloader module with yt-dlp integration" -``` - ---- - -## Task 5: Downloader Error Handling (TDD) - -**Files:** -- Modify: `tests/test_downloader.py` -- Modify: `src/downloader.py` - -**Step 1: Write test for download errors** - -Add to `tests/test_downloader.py`: - -```python -from src.downloader import DownloadError - - -def test_download_space_error_handling(tmp_path): - """Test that download errors are properly caught and wrapped.""" - space_url = "https://x.com/i/spaces/invalid" - - with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class: - mock_ytdl = Mock() - mock_ytdl_class.return_value.__enter__.return_value = mock_ytdl - mock_ytdl.extract_info.side_effect = Exception("Video unavailable") - - with pytest.raises(DownloadError) as exc_info: - download_space( - space_url=space_url, - output_dir=str(tmp_path), - cookies_file=None - ) - - assert "Failed to download" in str(exc_info.value) -``` - -**Step 2: Run test to verify it fails** - -Run: `pytest tests/test_downloader.py::test_download_space_error_handling -v` -Expected: FAIL with "cannot import name 'DownloadError'" - -**Step 3: Add error handling to downloader** - -Modify `src/downloader.py`: - -```python -import os -import yt_dlp -from typing import Optional - - -class DownloadError(Exception): - """Raised when download fails.""" - pass - - -def download_space( - space_url: str, - output_dir: str, - cookies_file: Optional[str] = None -) -> str: - """ - Downloads X Space recording using yt-dlp. - - Args: - space_url: URL of the X Space recording - output_dir: Directory to save the MP3 file - cookies_file: Optional path to cookies file for authentication - - Returns: - Path to the downloaded MP3 file - - Raises: - DownloadError: If download fails - """ - # Extract space ID from URL for filename - space_id = space_url.split('/')[-1] - - # Configure yt-dlp options - ydl_opts = { - 'format': 'bestaudio/best', - 'outtmpl': os.path.join(output_dir, f'space_{space_id}.%(ext)s'), - 'postprocessors': [{ - 'key': 'FFmpegExtractAudio', - 'preferredcodec': 'mp3', - }], - 'quiet': False, - 'no_warnings': False, - } - - # Add cookies if provided - if cookies_file: - ydl_opts['cookiefile'] = cookies_file - - # Download with yt-dlp - try: - with yt_dlp.YoutubeDL(ydl_opts) as ydl: - ydl.extract_info(space_url, download=True) - except Exception as e: - raise DownloadError(f"Failed to download Space: {str(e)}") - - # Return the expected output path - return os.path.join(output_dir, f'space_{space_id}.mp3') -``` - -**Step 4: Run test to verify it passes** - -Run: `pytest tests/test_downloader.py::test_download_space_error_handling -v` -Expected: PASS - -**Step 5: Run all downloader tests** - -Run: `pytest tests/test_downloader.py -v` -Expected: All tests PASS - -**Step 6: Commit** - -```bash -git add tests/test_downloader.py src/downloader.py -git commit -m "feat: add error handling to downloader" -``` - ---- - -## Task 6: CLI Main Module (TDD) - -**Files:** -- Create: `tests/test_main.py` -- Create: `src/main.py` - -**Step 1: Write test for CLI entry point** - -Create `tests/test_main.py`: - -```python -import pytest -from click.testing import CliRunner -from unittest.mock import patch, Mock -from src.main import cli - - -def test_cli_valid_url(tmp_path): - """Test CLI with valid URL.""" - runner = CliRunner() - - with patch('src.main.validate_space_url') as mock_validate, \ - patch('src.main.download_space') as mock_download: - - mock_validate.return_value = (True, "1ZkKzZWnVRRKv", None) - mock_download.return_value = str(tmp_path / "space_1ZkKzZWnVRRKv.mp3") - - result = runner.invoke(cli, ['https://x.com/i/spaces/1ZkKzZWnVRRKv']) - - assert result.exit_code == 0 - assert "Successfully downloaded" in result.output - assert "space_1ZkKzZWnVRRKv.mp3" in result.output -``` - -**Step 2: Run test to verify it fails** - -Run: `pytest tests/test_main.py::test_cli_valid_url -v` -Expected: FAIL with "cannot import name 'cli'" - -**Step 3: Write minimal CLI implementation** - -Create `src/main.py`: - -```python -import click -import os -from src.validator import validate_space_url -from src.downloader import download_space, DownloadError - - -@click.command() -@click.argument('url') -@click.option('--cookies', '-c', type=click.Path(exists=True), help='Path to cookies file') -@click.option('--output', '-o', type=click.Path(), default='.', help='Output directory') -def cli(url, cookies, output): - """ - Download X (Twitter) Space recordings as MP3 files. - - Example: - x-space-dl https://x.com/i/spaces/1ZkKzZWnVRRKv - """ - # Validate URL - is_valid, space_id, error = validate_space_url(url) - if not is_valid: - click.echo(f"Error: {error}", err=True) - raise click.Exit(1) - - # Download the Space - try: - click.echo(f"Downloading Space: {space_id}") - output_path = download_space(url, output, cookies) - click.echo(f"Successfully downloaded to: {output_path}") - except DownloadError as e: - click.echo(f"Error: {str(e)}", err=True) - raise click.Exit(2) - - -if __name__ == '__main__': - cli() -``` - -**Step 4: Run test to verify it passes** - -Run: `pytest tests/test_main.py::test_cli_valid_url -v` -Expected: PASS - -**Step 5: Commit** - -```bash -git add tests/test_main.py src/main.py -git commit -m "feat: add CLI main module with Click integration" -``` - ---- - -## Task 7: CLI Error Cases (TDD) - -**Files:** -- Modify: `tests/test_main.py` - -**Step 1: Write test for invalid URL** - -Add to `tests/test_main.py`: - -```python -def test_cli_invalid_url(): - """Test CLI with invalid URL.""" - runner = CliRunner() - - result = runner.invoke(cli, ['https://invalid.com/spaces/123']) - - assert result.exit_code == 1 - assert "Invalid URL format" in result.output -``` - -**Step 2: Run test to verify it passes** - -Run: `pytest tests/test_main.py::test_cli_invalid_url -v` -Expected: PASS (implementation already handles this) - -**Step 3: Write test for download failure** - -Add to `tests/test_main.py`: - -```python -def test_cli_download_failure(): - """Test CLI when download fails.""" - runner = CliRunner() - - with patch('src.main.validate_space_url') as mock_validate, \ - patch('src.main.download_space') as mock_download: - - mock_validate.return_value = (True, "1ZkKzZWnVRRKv", None) - mock_download.side_effect = DownloadError("Video unavailable") - - result = runner.invoke(cli, ['https://x.com/i/spaces/1ZkKzZWnVRRKv']) - - assert result.exit_code == 2 - assert "Video unavailable" in result.output -``` - -**Step 4: Run test to verify it passes** - -Run: `pytest tests/test_main.py::test_cli_download_failure -v` -Expected: PASS (implementation already handles this) - -**Step 5: Write test for cookies option** - -Add to `tests/test_main.py`: - -```python -def test_cli_with_cookies(tmp_path): - """Test CLI with cookies file.""" - runner = CliRunner() - - # Create a fake cookies file - cookies_file = tmp_path / "cookies.txt" - cookies_file.write_text("# Netscape HTTP Cookie File\n") - - with patch('src.main.validate_space_url') as mock_validate, \ - patch('src.main.download_space') as mock_download: - - mock_validate.return_value = (True, "1ZkKzZWnVRRKv", None) - mock_download.return_value = str(tmp_path / "space_1ZkKzZWnVRRKv.mp3") - - result = runner.invoke(cli, [ - 'https://x.com/i/spaces/1ZkKzZWnVRRKv', - '--cookies', str(cookies_file) - ]) - - assert result.exit_code == 0 - mock_download.assert_called_once() - assert mock_download.call_args[1]['cookies_file'] == str(cookies_file) -``` - -**Step 6: Run all CLI tests** - -Run: `pytest tests/test_main.py -v` -Expected: All tests PASS - -**Step 7: Commit** - -```bash -git add tests/test_main.py -git commit -m "test: add CLI error cases and cookies option tests" -``` - ---- - -## Task 8: README Documentation - -**Files:** -- Create: `README.md` - -**Step 1: Create comprehensive README** - -```markdown -# X Space Downloader - -A Python CLI tool that downloads X (Twitter) Space recordings and outputs them as MP3 files. - -## Features - -- Download X Space recordings as MP3 files -- Support for authenticated downloads using cookies -- Simple command-line interface -- Clear error messages - -## Installation - -### Prerequisites - -- Python 3.8 or higher -- ffmpeg (required by yt-dlp for audio conversion) - -Install ffmpeg: -```bash -# macOS -brew install ffmpeg - -# Ubuntu/Debian -sudo apt install ffmpeg - -# Windows -# Download from https://ffmpeg.org/download.html -``` - -### Install the tool - -```bash -git clone https://github.com/synapz-org/x-space-downloader.git -cd x-space-downloader -pip install -e . -``` - -## Usage - -### Basic usage - -```bash -x-space-dl https://x.com/i/spaces/1ZkKzZWnVRRKv -``` - -This will download the Space recording and save it as `space_1ZkKzZWnVRRKv.mp3` in the current directory. - -### With authentication (recommended) - -For better reliability and access to private Spaces, provide cookies from your logged-in X session: - -```bash -x-space-dl --cookies cookies.txt https://x.com/i/spaces/1ZkKzZWnVRRKv -``` - -### Specify output directory - -```bash -x-space-dl --output ~/Downloads https://x.com/i/spaces/1ZkKzZWnVRRKv -``` - -### Full options - -```bash -x-space-dl --help -``` - -## Exporting Cookies - -To download Spaces that require authentication, you need to export cookies from your logged-in X session. - -### Method 1: Browser Extension (Easiest) - -1. Install a cookie export extension: - - Chrome/Edge: [Get cookies.txt LOCALLY](https://chrome.google.com/webstore/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc) - - Firefox: [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/) - -2. Log in to X (twitter.com or x.com) -3. Click the extension icon and export cookies -4. Save as `cookies.txt` - -### Method 2: Manual Export (Advanced) - -1. Log in to X -2. Open Developer Tools (F12) -3. Go to Application/Storage → Cookies -4. Copy cookies in Netscape format to a file - -## Requirements - -- Python 3.8+ -- yt-dlp -- click -- ffmpeg (system dependency) - -## Supported URLs - -The tool supports X Space URLs in these formats: - -- `https://x.com/i/spaces/{SPACE_ID}` -- `https://twitter.com/i/spaces/{SPACE_ID}` - -**Note:** Only recordings are supported. Live Spaces must be recorded before downloading. - -## Error Messages - -| Error | Meaning | -|-------|---------| -| "Invalid URL format" | URL doesn't match X Space format | -| "Failed to download Space" | Recording not found or network error | -| "Authentication required" | Space requires login (use --cookies) | - -## Development - -### Running tests - -```bash -pip install pytest pytest-mock -pytest -``` - -### Project structure - -``` -x-space-downloader/ -├── src/ -│ ├── main.py # CLI entry point -│ ├── downloader.py # yt-dlp wrapper -│ └── validator.py # URL validation -├── tests/ -│ ├── test_main.py -│ ├── test_downloader.py -│ └── test_validator.py -└── docs/ - └── plans/ # Design documents -``` - -## License - -MIT License - See LICENSE file for details - -## Contributing - -Contributions are welcome! Please open an issue or submit a pull request. - -## Troubleshooting - -### "ffmpeg not found" -Install ffmpeg using your system's package manager (see Installation section). - -### "Video unavailable" -The Space recording may have been deleted or is private. Try using the --cookies option. - -### "Authentication required" -Export cookies from your logged-in X session and use the --cookies option. - -## Acknowledgments - -Built with: -- [yt-dlp](https://github.com/yt-dlp/yt-dlp) - Media downloader -- [Click](https://click.palletsprojects.com/) - CLI framework -``` - -**Step 2: Commit README** - -```bash -git add README.md -git commit -m "docs: add comprehensive README with usage and installation" -``` - ---- - -## Task 9: Development Dependencies - -**Files:** -- Create: `requirements-dev.txt` - -**Step 1: Create development requirements** - -```txt -pytest>=7.0.0 -pytest-mock>=3.10.0 -``` - -**Step 2: Commit** - -```bash -git add requirements-dev.txt -git commit -m "chore: add development dependencies" -``` - ---- - -## Task 10: Integration Test - -**Files:** -- Create: `tests/test_integration.py` - -**Step 1: Write integration test** - -Create `tests/test_integration.py`: - -```python -import pytest -from click.testing import CliRunner -from src.main import cli - - -@pytest.mark.integration -def test_full_workflow_with_mocks(tmp_path): - """Integration test for complete workflow.""" - runner = CliRunner() - - from unittest.mock import patch - - with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class: - # Mock successful download - mock_ytdl = mock_ytdl_class.return_value.__enter__.return_value - mock_ytdl.extract_info.return_value = {'id': '1ZkKzZWnVRRKv'} - - # Run CLI - result = runner.invoke(cli, [ - 'https://x.com/i/spaces/1ZkKzZWnVRRKv', - '--output', str(tmp_path) - ]) - - # Verify success - assert result.exit_code == 0 - assert "Successfully downloaded" in result.output - - # Verify yt-dlp was called correctly - mock_ytdl.extract_info.assert_called_once() - call_args = mock_ytdl_class.call_args - assert 'bestaudio' in call_args[0][0]['format'] - assert 'mp3' in call_args[0][0]['postprocessors'][0]['preferredcodec'] -``` - -**Step 2: Run integration test** - -Run: `pytest tests/test_integration.py -v` -Expected: PASS - -**Step 3: Run all tests** - -Run: `pytest -v` -Expected: All tests PASS - -**Step 4: Commit** - -```bash -git add tests/test_integration.py -git commit -m "test: add integration test for complete workflow" -``` - ---- - -## Task 11: Push to GitHub - -**Files:** -- N/A (git operations) - -**Step 1: Push all commits to GitHub** - -```bash -git push -u origin master -``` - -Expected: All commits pushed successfully - ---- - -## Verification Steps - -After completing all tasks: - -1. **Install the package:** - ```bash - pip install -e . - ``` - -2. **Run all tests:** - ```bash - pytest -v - ``` - Expected: All tests pass - -3. **Test CLI is installed:** - ```bash - x-space-dl --help - ``` - Expected: Help text displays - -4. **Test with invalid URL:** - ```bash - x-space-dl https://invalid.com/test - ``` - Expected: Error message about invalid URL format - -## Notes for Implementation - -- **TDD strictly:** Write test first, watch it fail, implement, watch it pass -- **Commit frequently:** After each passing test -- **YAGNI:** Don't add features not in the design (no GUI, no batch downloads, etc.) -- **DRY:** Reuse validator and downloader functions, don't duplicate logic -- **Error messages:** Keep them user-friendly and actionable -- **Testing:** Mock yt-dlp in tests to avoid actual network calls