mirror of
https://github.com/Dasemu/x-space-downloader.git
synced 2026-10-12 00:18:35 +00:00
chore: initial commit
This commit is contained in:
30
.gitignore
vendored
Normal file
30
.gitignore
vendored
Normal file
@@ -0,0 +1,30 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
.venv/
|
||||
ENV/
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
|
||||
# IDEs
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
# Downloads
|
||||
*.mp3
|
||||
*.m4a
|
||||
*.mp4
|
||||
cookies.txt
|
||||
202
README.md
Normal file
202
README.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# X Space Downloader
|
||||
|
||||
A Python CLI tool that downloads X (Twitter) Space recordings and renders an MP4 replay with audio and a compact participant view.
|
||||
|
||||
## Features
|
||||
|
||||
- Download X Space recordings as MP4 videos by default
|
||||
- Keep the original audio-only MP3 mode
|
||||
- Fetch participant metadata from X `AudioSpaceById` when cookies are provided
|
||||
- Download the replay chat timeline and highlight active speakers when cookies are provided
|
||||
- Accept an optional speaker timeline JSON file as a fallback/manual override
|
||||
- Support for authenticated downloads using cookies
|
||||
- Simple command-line interface
|
||||
- Clear error messages
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.8 or higher
|
||||
- ffmpeg (required for audio conversion and MP4 rendering)
|
||||
|
||||
Install ffmpeg:
|
||||
```bash
|
||||
# macOS
|
||||
brew install ffmpeg
|
||||
|
||||
# Ubuntu/Debian
|
||||
sudo apt install ffmpeg
|
||||
|
||||
# Windows
|
||||
# Download from https://ffmpeg.org/download.html
|
||||
```
|
||||
|
||||
### Install the tool
|
||||
|
||||
```bash
|
||||
git clone https://github.com/synapz-org/x-space-downloader.git
|
||||
cd x-space-downloader
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic usage
|
||||
|
||||
```bash
|
||||
x-space-dl https://x.com/i/spaces/1ZkKzZWnVRRKv
|
||||
```
|
||||
|
||||
This will download the Space recording and save it as `space_1ZkKzZWnVRRKv.mp4` in the current directory.
|
||||
|
||||
### Audio-only mode
|
||||
|
||||
```bash
|
||||
x-space-dl --mode audio https://x.com/i/spaces/1ZkKzZWnVRRKv
|
||||
```
|
||||
|
||||
This keeps the original behavior and writes `space_1ZkKzZWnVRRKv.mp3`.
|
||||
|
||||
### With authentication (recommended)
|
||||
|
||||
For better reliability and access to private Spaces, provide cookies from your logged-in X session:
|
||||
|
||||
```bash
|
||||
x-space-dl --cookies cookies.txt https://x.com/i/spaces/1ZkKzZWnVRRKv
|
||||
```
|
||||
|
||||
Cookies are also used to fetch the Space title, host, speakers, avatars, and replay chat timeline from X/Periscope metadata.
|
||||
|
||||
### Speaker timeline
|
||||
|
||||
With cookies, the tool automatically calls the replay `history` endpoint and builds participant/speaker timing from Space microphone events. If you already have speaker intervals from another extractor, you can override that with JSON:
|
||||
|
||||
```bash
|
||||
x-space-dl --cookies cookies.txt --speaker-timeline speakers.json https://x.com/i/spaces/1ZkKzZWnVRRKv
|
||||
```
|
||||
|
||||
Supported JSON shape:
|
||||
|
||||
```json
|
||||
[
|
||||
{"start": 0.0, "end": 12.4, "speaker": "alice"},
|
||||
{"start": 12.4, "end": 31.0, "speaker": "bob"}
|
||||
]
|
||||
```
|
||||
|
||||
The `speaker` value should match the X handle without requiring `@`.
|
||||
|
||||
### Specify output directory
|
||||
|
||||
```bash
|
||||
x-space-dl --output ~/Downloads https://x.com/i/spaces/1ZkKzZWnVRRKv
|
||||
```
|
||||
|
||||
### Full options
|
||||
|
||||
```bash
|
||||
x-space-dl --help
|
||||
```
|
||||
|
||||
## Exporting Cookies
|
||||
|
||||
To download Spaces that require authentication, you need to export cookies from your logged-in X session.
|
||||
|
||||
### Method 1: Browser Extension (Easiest)
|
||||
|
||||
1. Install a cookie export extension:
|
||||
- Chrome/Edge: [Get cookies.txt LOCALLY](https://chrome.google.com/webstore/detail/get-cookiestxt-locally/cclelndahbckbenkjhflpdbgdldlbecc)
|
||||
- Firefox: [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/)
|
||||
|
||||
2. Log in to X (twitter.com or x.com)
|
||||
3. Click the extension icon and export cookies
|
||||
4. Save as `cookies.txt`
|
||||
|
||||
### Method 2: Manual Export (Advanced)
|
||||
|
||||
1. Log in to X
|
||||
2. Open Developer Tools (F12)
|
||||
3. Go to Application/Storage → Cookies
|
||||
4. Copy cookies in Netscape format to a file
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.8+
|
||||
- yt-dlp
|
||||
- click
|
||||
- requests
|
||||
- Pillow
|
||||
- ffmpeg (system dependency)
|
||||
|
||||
## Supported URLs
|
||||
|
||||
The tool supports X Space URLs in these formats:
|
||||
|
||||
- `https://x.com/i/spaces/{SPACE_ID}`
|
||||
- `https://twitter.com/i/spaces/{SPACE_ID}`
|
||||
|
||||
**Note:** Only recordings are supported. Live Spaces must be recorded before downloading.
|
||||
|
||||
## Error Messages
|
||||
|
||||
| Error | Meaning |
|
||||
|-------|---------|
|
||||
| "Invalid URL format" | URL doesn't match X Space format |
|
||||
| "Failed to download Space" | Recording not found or network error |
|
||||
| "Authentication required" | Space requires login (use --cookies) |
|
||||
|
||||
## Development
|
||||
|
||||
### Running tests
|
||||
|
||||
```bash
|
||||
pip install pytest pytest-mock
|
||||
pytest
|
||||
```
|
||||
|
||||
### Project structure
|
||||
|
||||
```
|
||||
x-space-downloader/
|
||||
├── src/
|
||||
│ ├── main.py # CLI entry point
|
||||
│ ├── downloader.py # yt-dlp wrapper
|
||||
│ ├── x_api.py # X GraphQL metadata client
|
||||
│ ├── video.py # MP4 participant-view renderer
|
||||
│ ├── models.py # Shared data models
|
||||
│ └── validator.py # URL validation
|
||||
├── tests/
|
||||
│ ├── test_main.py
|
||||
│ ├── test_downloader.py
|
||||
│ ├── test_video.py
|
||||
│ ├── test_x_api.py
|
||||
│ └── test_validator.py
|
||||
└── docs/
|
||||
└── plans/ # Design documents
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT License - See LICENSE file for details
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please open an issue or submit a pull request.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "ffmpeg not found"
|
||||
Install ffmpeg using your system's package manager (see Installation section).
|
||||
|
||||
### "Video unavailable"
|
||||
The Space recording may have been deleted or is private. Try using the --cookies option.
|
||||
|
||||
### "Authentication required"
|
||||
Export cookies from your logged-in X session and use the --cookies option.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Built with:
|
||||
- [yt-dlp](https://github.com/yt-dlp/yt-dlp) - Media downloader
|
||||
- [Click](https://click.palletsprojects.com/) - CLI framework
|
||||
188
docs/plans/2025-10-26-x-space-downloader-design.md
Normal file
188
docs/plans/2025-10-26-x-space-downloader-design.md
Normal file
@@ -0,0 +1,188 @@
|
||||
# X Space Downloader - Design Document
|
||||
|
||||
**Date:** 2025-10-26
|
||||
**Status:** Approved
|
||||
|
||||
## Overview
|
||||
|
||||
A Python CLI tool that downloads X (Twitter) Space recordings and outputs them as MP3 files. Takes a Space URL as input and produces an audio file.
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Input:** X Space recording URL (e.g., `https://x.com/i/spaces/1ZkKzZWnVRRKv`)
|
||||
- **Output:** MP3 audio file
|
||||
- **Interface:** Command-line tool
|
||||
- **Authentication:** Supports X account credentials via cookies
|
||||
- **Scope:** Recordings only (not live Spaces)
|
||||
- **Error Handling:** Clear error messages for invalid or non-recording URLs
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Technology Choices
|
||||
|
||||
- **Language:** Python
|
||||
- **Core Engine:** yt-dlp (handles X authentication and media extraction)
|
||||
- **CLI Framework:** Click (argument parsing and user interface)
|
||||
- **Pattern:** Thin wrapper around yt-dlp
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
x-space-downloader/
|
||||
├── src/
|
||||
│ ├── __init__.py
|
||||
│ ├── main.py # CLI entry point with argument parsing
|
||||
│ ├── downloader.py # Wraps yt-dlp functionality
|
||||
│ └── validator.py # URL validation and error handling
|
||||
├── tests/
|
||||
│ ├── test_validator.py
|
||||
│ └── test_downloader.py
|
||||
├── requirements.txt # Dependencies: yt-dlp, click
|
||||
├── setup.py # Makes it installable
|
||||
└── README.md
|
||||
```
|
||||
|
||||
### User Workflow
|
||||
|
||||
```bash
|
||||
x-space-dl --cookies cookies.txt https://x.com/i/spaces/1ZkKzZWnVRRKv
|
||||
# Output: space_1ZkKzZWnVRRKv.mp3
|
||||
```
|
||||
|
||||
## Component Design
|
||||
|
||||
### validator.py
|
||||
|
||||
**Responsibilities:**
|
||||
- Validate URL format (must match X Spaces pattern: `https://x.com/i/spaces/{SPACE_ID}`)
|
||||
- Check if URL points to a recording vs live Space
|
||||
- Return clear error messages for invalid inputs
|
||||
|
||||
**Key Interface:**
|
||||
```python
|
||||
def validate_space_url(url: str) -> Tuple[bool, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Validates X Space URL.
|
||||
|
||||
Returns:
|
||||
(is_valid, space_id, error_message)
|
||||
"""
|
||||
```
|
||||
|
||||
### downloader.py
|
||||
|
||||
**Responsibilities:**
|
||||
- Configure yt-dlp options (format: mp3, quality, output template)
|
||||
- Pass cookie file to yt-dlp for authentication
|
||||
- Translate yt-dlp output/errors to user-friendly messages
|
||||
- Show download progress
|
||||
|
||||
**Key Interface:**
|
||||
```python
|
||||
def download_space(
|
||||
space_url: str,
|
||||
output_dir: str,
|
||||
cookies_file: Optional[str]
|
||||
) -> str:
|
||||
"""
|
||||
Downloads Space recording using yt-dlp.
|
||||
|
||||
Returns:
|
||||
Path to downloaded MP3 file
|
||||
"""
|
||||
```
|
||||
|
||||
**yt-dlp Configuration:**
|
||||
- Output format: mp3 (audio only)
|
||||
- Output template: `space_{space_id}.mp3`
|
||||
- Post-processing: convert to mp3 if needed
|
||||
- Progress hooks for user feedback
|
||||
|
||||
### main.py
|
||||
|
||||
**Responsibilities:**
|
||||
- CLI argument parsing with Click
|
||||
- Orchestrate validation → download flow
|
||||
- Handle and display errors appropriately
|
||||
- Exit with appropriate status codes
|
||||
|
||||
**CLI Arguments:**
|
||||
- `URL` (required): X Space URL
|
||||
- `--cookies` (optional): Path to cookies file
|
||||
- `--output` (optional): Output directory (default: current directory)
|
||||
|
||||
**Exit Codes:**
|
||||
- `0`: Success
|
||||
- `1`: Invalid URL
|
||||
- `2`: Download failed
|
||||
- `3`: Authentication required
|
||||
|
||||
## Data Flow
|
||||
|
||||
1. User provides URL + optional cookies file
|
||||
2. **Validator** checks URL format → extracts space_id
|
||||
3. **Downloader** configures yt-dlp with:
|
||||
- Output format: mp3 (audio only)
|
||||
- Output template: `space_{space_id}.mp3`
|
||||
- Cookies file if provided
|
||||
- Post-processing: convert to mp3 if needed
|
||||
4. yt-dlp fetches metadata, downloads audio stream, converts to mp3
|
||||
5. Return file path to user
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error Scenario | Detection | User Message |
|
||||
|----------------|-----------|--------------|
|
||||
| Invalid URL format | Validator regex check | "Invalid URL format. Expected: https://x.com/i/spaces/{SPACE_ID}" |
|
||||
| Space doesn't exist | yt-dlp 404 error | "Recording not found. Check URL or Space may be deleted." |
|
||||
| Authentication required | yt-dlp auth error | "Authentication required. Please provide cookies file with --cookies" |
|
||||
| Network errors | yt-dlp timeout/connection | "Network error. Check connection and try again." |
|
||||
| Space is live | Validator metadata check | "Error: This is a live Space, not a recording." |
|
||||
| Disk space/permissions | File system errors | "Cannot write to output directory. Check permissions and disk space." |
|
||||
|
||||
## Authentication
|
||||
|
||||
**Cookie-based Authentication:**
|
||||
- User exports cookies from logged-in X session
|
||||
- Provides cookie file path via `--cookies` flag
|
||||
- yt-dlp uses cookies for authenticated requests
|
||||
|
||||
**Cookie Export Instructions:**
|
||||
- Browser extensions: "Get cookies.txt", "cookies.txt"
|
||||
- Manual: Export via browser DevTools
|
||||
- Instructions included in README
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Validator: Test various URL formats, valid/invalid patterns
|
||||
- Error message generation
|
||||
|
||||
**Integration Tests:**
|
||||
- Full download flow with yt-dlp
|
||||
- Use test Space URLs or mock yt-dlp calls
|
||||
- Cookie authentication flow
|
||||
|
||||
**Error Scenario Tests:**
|
||||
- Invalid URLs
|
||||
- Missing cookies
|
||||
- Network failures
|
||||
- Live Space detection
|
||||
|
||||
## Dependencies
|
||||
|
||||
**Core:**
|
||||
- `yt-dlp`: Media downloading and extraction
|
||||
- `click`: CLI framework
|
||||
|
||||
**Development:**
|
||||
- `pytest`: Testing framework
|
||||
- `pytest-mock`: Mocking for tests
|
||||
|
||||
## Future Considerations (Out of Scope)
|
||||
|
||||
- Live Space recording
|
||||
- Batch downloading multiple Spaces
|
||||
- GUI interface
|
||||
- Format options beyond MP3
|
||||
- Metadata extraction (speakers, title, description)
|
||||
3
pytest.ini
Normal file
3
pytest.ini
Normal file
@@ -0,0 +1,3 @@
|
||||
[pytest]
|
||||
markers =
|
||||
integration: marks tests as integration tests (deselect with '-m "not integration"')
|
||||
2
requirements-dev.txt
Normal file
2
requirements-dev.txt
Normal file
@@ -0,0 +1,2 @@
|
||||
pytest>=7.0.0
|
||||
pytest-mock>=3.10.0
|
||||
4
requirements.txt
Normal file
4
requirements.txt
Normal file
@@ -0,0 +1,4 @@
|
||||
yt-dlp>=2024.0.0
|
||||
click>=8.0.0
|
||||
requests>=2.31.0
|
||||
Pillow>=10.0.0
|
||||
19
setup.py
Normal file
19
setup.py
Normal file
@@ -0,0 +1,19 @@
|
||||
from setuptools import setup, find_packages
|
||||
|
||||
setup(
|
||||
name="x-space-downloader",
|
||||
version="0.1.0",
|
||||
packages=find_packages(),
|
||||
install_requires=[
|
||||
"yt-dlp>=2024.0.0",
|
||||
"click>=8.0.0",
|
||||
"requests>=2.31.0",
|
||||
"Pillow>=10.0.0",
|
||||
],
|
||||
entry_points={
|
||||
"console_scripts": [
|
||||
"x-space-dl=src.main:cli",
|
||||
],
|
||||
},
|
||||
python_requires=">=3.8",
|
||||
)
|
||||
0
src/__init__.py
Normal file
0
src/__init__.py
Normal file
163
src/downloader.py
Normal file
163
src/downloader.py
Normal file
@@ -0,0 +1,163 @@
|
||||
import os
|
||||
from glob import glob
|
||||
import yt_dlp
|
||||
from typing import Optional
|
||||
from src.validator import validate_space_url
|
||||
from src.models import SpaceMetadata
|
||||
from src.video import RenderError, SpeakerTimeline, render_space_video
|
||||
from src.x_api import XApiClient, XApiError
|
||||
|
||||
|
||||
class DownloadError(Exception):
|
||||
"""Raised when download fails."""
|
||||
pass
|
||||
|
||||
|
||||
def download_space(
|
||||
space_url: str,
|
||||
output_dir: str,
|
||||
cookies_file: Optional[str] = None,
|
||||
audio_format: str = "mp3",
|
||||
) -> str:
|
||||
"""
|
||||
Downloads X Space recording using yt-dlp.
|
||||
|
||||
Args:
|
||||
space_url: URL of the X Space recording
|
||||
output_dir: Directory to save the MP3 file
|
||||
cookies_file: Optional path to cookies file for authentication
|
||||
audio_format: Audio container/codec to write
|
||||
|
||||
Returns:
|
||||
Path to the downloaded MP3 file
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
# Extract space ID from URL using validator for robustness
|
||||
is_valid, space_id, error_message = validate_space_url(space_url)
|
||||
if not is_valid:
|
||||
raise ValueError(f"Invalid space URL: {error_message}")
|
||||
|
||||
if space_id is None:
|
||||
raise ValueError("Could not extract space ID from URL")
|
||||
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
# Configure yt-dlp options
|
||||
ydl_opts = {
|
||||
'format': 'bestaudio/best',
|
||||
'outtmpl': os.path.join(output_dir, f'space_{space_id}.%(ext)s'),
|
||||
'quiet': False,
|
||||
'no_warnings': False,
|
||||
}
|
||||
if audio_format != "native":
|
||||
ydl_opts['postprocessors'] = [{
|
||||
'key': 'FFmpegExtractAudio',
|
||||
'preferredcodec': audio_format,
|
||||
}]
|
||||
|
||||
# Add cookies if provided
|
||||
if cookies_file:
|
||||
ydl_opts['cookiefile'] = cookies_file
|
||||
|
||||
# Download with yt-dlp
|
||||
try:
|
||||
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
|
||||
ydl.extract_info(space_url, download=True)
|
||||
except Exception as e:
|
||||
raise DownloadError(f"Failed to download Space: {str(e)}")
|
||||
|
||||
# Return the expected output path
|
||||
if audio_format == "native":
|
||||
return os.path.join(output_dir, f'space_{space_id}')
|
||||
return os.path.join(output_dir, f'space_{space_id}.{audio_format}')
|
||||
|
||||
|
||||
def download_space_video(
|
||||
space_url: str,
|
||||
output_dir: str,
|
||||
cookies_file: Optional[str] = None,
|
||||
speaker_timeline_file: Optional[str] = None,
|
||||
max_duration: Optional[float] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download an X Space replay and render an MP4 participant view with audio.
|
||||
"""
|
||||
is_valid, space_id, error_message = validate_space_url(space_url)
|
||||
if not is_valid:
|
||||
raise ValueError(f"Invalid space URL: {error_message}")
|
||||
if space_id is None:
|
||||
raise ValueError("Could not extract space ID from URL")
|
||||
|
||||
metadata = fetch_required_metadata(space_id, cookies_file)
|
||||
timeline = SpeakerTimeline.from_file(speaker_timeline_file) if speaker_timeline_file else None
|
||||
if timeline is None:
|
||||
timeline = fetch_required_timeline(metadata, cookies_file)
|
||||
|
||||
audio_path = download_space(space_url, output_dir, cookies_file, audio_format="native")
|
||||
if not os.path.exists(audio_path):
|
||||
candidates = sorted(glob(os.path.join(output_dir, f"space_{space_id}.*")))
|
||||
candidates = [path for path in candidates if not path.endswith(".mp4")]
|
||||
if not candidates:
|
||||
raise DownloadError("Downloaded audio file was not found")
|
||||
audio_path = candidates[0]
|
||||
output_path = os.path.join(output_dir, f"space_{space_id}.mp4")
|
||||
|
||||
try:
|
||||
return render_space_video(
|
||||
audio_path=audio_path,
|
||||
output_path=output_path,
|
||||
metadata=metadata,
|
||||
timeline=timeline,
|
||||
max_duration=max_duration,
|
||||
)
|
||||
except RenderError as exc:
|
||||
raise DownloadError(f"Failed to render Space video: {exc}") from exc
|
||||
|
||||
|
||||
def fetch_metadata_or_fallback(space_id: str, cookies_file: Optional[str]) -> SpaceMetadata:
|
||||
try:
|
||||
return XApiClient(cookies_file=cookies_file).fetch_space_metadata(space_id)
|
||||
except XApiError:
|
||||
return SpaceMetadata(space_id=space_id, title=f"X Space {space_id}", participants=[])
|
||||
|
||||
|
||||
def fetch_required_metadata(space_id: str, cookies_file: Optional[str]) -> SpaceMetadata:
|
||||
try:
|
||||
return XApiClient(cookies_file=cookies_file).fetch_space_metadata(space_id)
|
||||
except XApiError as exc:
|
||||
hint = " Export logged-in X cookies and pass them with --cookies." if not cookies_file else ""
|
||||
raise DownloadError(f"Failed to fetch Space metadata.{hint} Details: {exc}") from exc
|
||||
|
||||
|
||||
def fetch_required_timeline(metadata: SpaceMetadata, cookies_file: Optional[str]) -> SpeakerTimeline:
|
||||
if not cookies_file:
|
||||
raise DownloadError("Video mode requires --cookies to fetch the replay participant timeline.")
|
||||
try:
|
||||
intervals, first_seen, history_participants = XApiClient(cookies_file=cookies_file).fetch_space_timeline(metadata)
|
||||
except XApiError as exc:
|
||||
raise DownloadError(f"Failed to fetch Space replay timeline: {exc}") from exc
|
||||
merge_history_participants(metadata, history_participants)
|
||||
if not intervals and not first_seen:
|
||||
raise DownloadError("Space replay timeline did not include participant or microphone events.")
|
||||
return SpeakerTimeline(intervals, first_seen)
|
||||
|
||||
|
||||
def fetch_timeline_or_none(metadata: SpaceMetadata, cookies_file: str) -> Optional[SpeakerTimeline]:
|
||||
try:
|
||||
intervals, first_seen, history_participants = XApiClient(cookies_file=cookies_file).fetch_space_timeline(metadata)
|
||||
except XApiError:
|
||||
return None
|
||||
merge_history_participants(metadata, history_participants)
|
||||
if not intervals and not first_seen:
|
||||
return None
|
||||
return SpeakerTimeline(intervals, first_seen)
|
||||
|
||||
|
||||
def merge_history_participants(metadata: SpaceMetadata, history_participants) -> None:
|
||||
existing = {participant.username.lower() for participant in metadata.participants}
|
||||
for participant in history_participants:
|
||||
if participant.username.lower() not in existing:
|
||||
metadata.participants.append(participant)
|
||||
existing.add(participant.username.lower())
|
||||
56
src/main.py
Normal file
56
src/main.py
Normal file
@@ -0,0 +1,56 @@
|
||||
import click
|
||||
from click.exceptions import Exit
|
||||
from src.validator import validate_space_url
|
||||
from src.downloader import download_space, download_space_video, DownloadError
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument('url')
|
||||
@click.option('--cookies', '-c', type=click.Path(exists=True), help='Path to cookies file')
|
||||
@click.option('--output', '-o', type=click.Path(), default='.', help='Output directory')
|
||||
@click.option(
|
||||
'--mode',
|
||||
type=click.Choice(['audio', 'video']),
|
||||
default='video',
|
||||
show_default=True,
|
||||
help='Output type to generate',
|
||||
)
|
||||
@click.option(
|
||||
'--speaker-timeline',
|
||||
type=click.Path(exists=True),
|
||||
help='JSON file with speaker intervals: [{"start": 0, "end": 5, "speaker": "handle"}]',
|
||||
)
|
||||
@click.option(
|
||||
'--max-duration',
|
||||
type=float,
|
||||
hidden=True,
|
||||
help='Render only the first N seconds; intended for tests and previews',
|
||||
)
|
||||
def cli(url, cookies, output, mode, speaker_timeline, max_duration):
|
||||
"""
|
||||
Download X (Twitter) Space recordings as MP3 or MP4 files.
|
||||
|
||||
Example:
|
||||
x-space-dl https://x.com/i/spaces/1ZkKzZWnVRRKv
|
||||
"""
|
||||
# Validate URL
|
||||
is_valid, space_id, error = validate_space_url(url)
|
||||
if not is_valid:
|
||||
click.echo(f"Error: {error}", err=True)
|
||||
raise Exit(1)
|
||||
|
||||
# Download the Space
|
||||
try:
|
||||
click.echo(f"Downloading Space: {space_id}")
|
||||
if mode == 'audio':
|
||||
output_path = download_space(url, output, cookies)
|
||||
else:
|
||||
output_path = download_space_video(url, output, cookies, speaker_timeline, max_duration)
|
||||
click.echo(f"Successfully downloaded to: {output_path}")
|
||||
except DownloadError as e:
|
||||
click.echo(f"Error: {str(e)}", err=True)
|
||||
raise Exit(2)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
cli()
|
||||
34
src/models.py
Normal file
34
src/models.py
Normal file
@@ -0,0 +1,34 @@
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Participant:
|
||||
"""A user visible in the Space participant tray."""
|
||||
|
||||
user_id: str
|
||||
display_name: str
|
||||
username: str
|
||||
avatar_url: Optional[str] = None
|
||||
role: str = "speaker"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SpaceMetadata:
|
||||
"""Metadata needed to render a Space replay video."""
|
||||
|
||||
space_id: str
|
||||
title: str
|
||||
participants: List[Participant] = field(default_factory=list)
|
||||
media_key: Optional[str] = None
|
||||
started_at_ms: Optional[int] = None
|
||||
ended_at_ms: Optional[int] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SpeakerInterval:
|
||||
"""An interval where a known participant is speaking."""
|
||||
|
||||
start: float
|
||||
end: float
|
||||
speaker: str
|
||||
23
src/validator.py
Normal file
23
src/validator.py
Normal file
@@ -0,0 +1,23 @@
|
||||
import re
|
||||
from typing import Tuple, Optional
|
||||
|
||||
|
||||
def validate_space_url(url: str) -> Tuple[bool, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Validates X Space URL format.
|
||||
|
||||
Args:
|
||||
url: The URL to validate
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, space_id, error_message)
|
||||
"""
|
||||
# Pattern for X Space URLs (supports both x.com and twitter.com)
|
||||
pattern = r"https://(x|twitter)\.com/i/spaces/([a-zA-Z0-9]+)"
|
||||
match = re.match(pattern, url)
|
||||
|
||||
if not match:
|
||||
return False, None, "Invalid URL format. Expected: https://x.com/i/spaces/{SPACE_ID}"
|
||||
|
||||
space_id = match.group(2) # Changed from group(1) to group(2)
|
||||
return True, space_id, None
|
||||
464
src/video.py
Normal file
464
src/video.py
Normal file
@@ -0,0 +1,464 @@
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import replace
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from typing import Dict, Iterable, List, Optional, Set, Tuple
|
||||
|
||||
import requests
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
from src.models import Participant, SpaceMetadata, SpeakerInterval
|
||||
|
||||
|
||||
class RenderError(Exception):
|
||||
"""Raised when the MP4 render fails."""
|
||||
|
||||
|
||||
class SpeakerTimeline:
|
||||
"""Lookup table for active speakers during a replay."""
|
||||
|
||||
def __init__(self, intervals: Iterable[SpeakerInterval], participant_first_seen: Optional[Dict[str, float]] = None):
|
||||
self.intervals = sorted(intervals, key=lambda item: item.start)
|
||||
self.participant_first_seen = {
|
||||
username.lstrip("@").lower(): float(seconds)
|
||||
for username, seconds in (participant_first_seen or {}).items()
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_file(cls, path: str) -> "SpeakerTimeline":
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
|
||||
return cls.from_file_data(payload)
|
||||
|
||||
@classmethod
|
||||
def from_file_data(cls, payload) -> "SpeakerTimeline":
|
||||
rows = payload.get("speakers", payload) if isinstance(payload, dict) else payload
|
||||
intervals = []
|
||||
for row in rows:
|
||||
intervals.append(
|
||||
SpeakerInterval(
|
||||
start=float(row["start"]),
|
||||
end=float(row["end"]),
|
||||
speaker=str(row.get("speaker") or row.get("username") or row.get("handle")),
|
||||
)
|
||||
)
|
||||
return cls(intervals)
|
||||
|
||||
def active_at(self, seconds: float) -> Optional[str]:
|
||||
active = self.active_speakers_at(seconds)
|
||||
return active[0] if active else None
|
||||
|
||||
def active_speakers_at(self, seconds: float) -> List[str]:
|
||||
return [
|
||||
interval.speaker.lstrip("@").lower()
|
||||
for interval in self.intervals
|
||||
if interval.start <= seconds < interval.end
|
||||
]
|
||||
|
||||
def first_seen_at(self, username: str) -> Optional[float]:
|
||||
return self.participant_first_seen.get(username.lstrip("@").lower())
|
||||
|
||||
|
||||
def render_space_video(
|
||||
audio_path: str,
|
||||
output_path: str,
|
||||
metadata: SpaceMetadata,
|
||||
timeline: Optional[SpeakerTimeline] = None,
|
||||
width: int = 720,
|
||||
height: int = 720,
|
||||
fps: int = 1,
|
||||
max_duration: Optional[float] = None,
|
||||
) -> str:
|
||||
"""Render an MP4 that mimics the compact X Spaces participant view."""
|
||||
if not os.path.exists(audio_path):
|
||||
raise RenderError(f"Audio file not found: {audio_path}")
|
||||
if shutil.which("ffmpeg") is None:
|
||||
raise RenderError("ffmpeg is required to render the video")
|
||||
|
||||
duration = max_duration or probe_duration(audio_path)
|
||||
if duration <= 0:
|
||||
raise RenderError("Could not determine audio duration")
|
||||
|
||||
output = Path(output_path)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
avatars = load_avatars(metadata.participants, avatar_size=168)
|
||||
|
||||
if timeline is None:
|
||||
frame = render_frame(metadata, avatars, None, 0, width, height)
|
||||
return render_looped_image_video(frame, audio_path, str(output), duration)
|
||||
|
||||
command = build_ffmpeg_command(audio_path, str(output), width, height, fps)
|
||||
|
||||
process = subprocess.Popen(command, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
total_frames = max(1, int(math.ceil(duration * fps)))
|
||||
frame_cache: Dict[Tuple[Tuple[str, ...], Optional[str], int], bytes] = {}
|
||||
for frame_index in range(total_frames):
|
||||
seconds = frame_index / fps
|
||||
participants = visible_participants(metadata.participants, timeline, seconds)
|
||||
active = tuple(timeline.active_speakers_at(seconds)) if timeline else ()
|
||||
cache_key = (tuple(participant.username.lower() for participant in participants), active)
|
||||
frame_bytes = frame_cache.get(cache_key)
|
||||
if frame_bytes is None:
|
||||
frame = render_frame(
|
||||
replace(metadata, participants=participants),
|
||||
avatars,
|
||||
timeline,
|
||||
seconds,
|
||||
width,
|
||||
height,
|
||||
participants_are_filtered=True,
|
||||
)
|
||||
frame_bytes = frame.tobytes()
|
||||
frame_cache[cache_key] = frame_bytes
|
||||
process.stdin.write(frame_bytes)
|
||||
except BrokenPipeError as exc:
|
||||
raise RenderError("ffmpeg closed before all frames were written") from exc
|
||||
finally:
|
||||
if process.stdin:
|
||||
process.stdin.close()
|
||||
|
||||
stderr = process.stderr.read().decode("utf-8", errors="replace") if process.stderr else ""
|
||||
return_code = process.wait()
|
||||
if return_code != 0:
|
||||
raise RenderError(stderr.strip() or "ffmpeg failed")
|
||||
|
||||
return str(output)
|
||||
|
||||
|
||||
def render_looped_image_video(frame: Image.Image, audio_path: str, output_path: str, duration: float) -> str:
|
||||
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as handle:
|
||||
frame_path = handle.name
|
||||
try:
|
||||
frame.save(frame_path)
|
||||
command = [
|
||||
"ffmpeg",
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-loop",
|
||||
"1",
|
||||
"-framerate",
|
||||
"1",
|
||||
"-i",
|
||||
frame_path,
|
||||
"-i",
|
||||
audio_path,
|
||||
"-t",
|
||||
str(duration),
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-preset",
|
||||
"veryfast",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-c:a",
|
||||
"copy",
|
||||
"-shortest",
|
||||
output_path,
|
||||
]
|
||||
result = subprocess.run(command, capture_output=True, text=True, check=False)
|
||||
if result.returncode != 0:
|
||||
raise RenderError(result.stderr.strip() or "ffmpeg failed")
|
||||
return output_path
|
||||
finally:
|
||||
try:
|
||||
os.unlink(frame_path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def build_ffmpeg_command(audio_path: str, output_path: str, width: int, height: int, fps: int) -> List[str]:
|
||||
return [
|
||||
"ffmpeg",
|
||||
"-y",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-vcodec",
|
||||
"rawvideo",
|
||||
"-s",
|
||||
f"{width}x{height}",
|
||||
"-pix_fmt",
|
||||
"rgb24",
|
||||
"-r",
|
||||
str(fps),
|
||||
"-i",
|
||||
"-",
|
||||
"-i",
|
||||
audio_path,
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-preset",
|
||||
"veryfast",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-c:a",
|
||||
"copy",
|
||||
"-shortest",
|
||||
output_path,
|
||||
]
|
||||
|
||||
|
||||
def probe_duration(path: str) -> float:
|
||||
if shutil.which("ffprobe") is None:
|
||||
raise RenderError("ffprobe is required to inspect audio duration")
|
||||
|
||||
command = [
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-show_entries",
|
||||
"format=duration",
|
||||
"-of",
|
||||
"default=noprint_wrappers=1:nokey=1",
|
||||
path,
|
||||
]
|
||||
result = subprocess.run(command, capture_output=True, text=True, check=False)
|
||||
if result.returncode != 0:
|
||||
raise RenderError(result.stderr.strip() or "ffprobe failed")
|
||||
return float(result.stdout.strip())
|
||||
|
||||
|
||||
def load_avatars(participants: List[Participant], avatar_size: int) -> Dict[str, Image.Image]:
|
||||
avatars: Dict[str, Image.Image] = {}
|
||||
for participant in participants:
|
||||
image = None
|
||||
if participant.avatar_url:
|
||||
try:
|
||||
response = requests.get(participant.avatar_url, timeout=8)
|
||||
response.raise_for_status()
|
||||
image = Image.open(BytesIO(response.content)).convert("RGB")
|
||||
except Exception:
|
||||
image = None
|
||||
|
||||
if image is None:
|
||||
image = placeholder_avatar(participant, avatar_size)
|
||||
avatars[participant.username.lower()] = crop_circle(image, avatar_size)
|
||||
return avatars
|
||||
|
||||
|
||||
def render_frame(
|
||||
metadata: SpaceMetadata,
|
||||
avatars: Dict[str, Image.Image],
|
||||
timeline: Optional[SpeakerTimeline],
|
||||
seconds: float,
|
||||
width: int,
|
||||
height: int,
|
||||
participants_are_filtered: bool = False,
|
||||
) -> Image.Image:
|
||||
participants = metadata.participants if participants_are_filtered else visible_participants(metadata.participants, timeline, seconds)
|
||||
active = set(timeline.active_speakers_at(seconds)) if timeline else set()
|
||||
|
||||
frame = Image.new("RGB", (width, height), (18, 21, 24))
|
||||
draw = ImageDraw.Draw(frame)
|
||||
fonts = load_fonts()
|
||||
|
||||
draw_rounded_rectangle(draw, (24, 24, width - 24, height - 24), 28, fill=(9, 13, 17), outline=(39, 46, 54))
|
||||
draw.text((54, 54), truncate(metadata.title, 42), fill=(238, 242, 246), font=fonts["title"])
|
||||
subtitle = "X Space replay"
|
||||
if timeline and timeline.intervals:
|
||||
subtitle += " · speaker timeline"
|
||||
draw.text((54, 92), subtitle, fill=(139, 152, 165), font=fonts["small"])
|
||||
|
||||
if not participants:
|
||||
participants = [Participant("space", "Space audio", "space", role="speaker")]
|
||||
|
||||
layout = participant_layout(width, height, len(participants))
|
||||
for participant, box in zip(participants, layout):
|
||||
draw_participant(frame, draw, participant, avatars.get(participant.username.lower()), box, active, seconds, fonts)
|
||||
|
||||
return frame
|
||||
|
||||
|
||||
def visible_participants(
|
||||
participants: List[Participant],
|
||||
timeline: Optional[SpeakerTimeline] = None,
|
||||
seconds: float = 0,
|
||||
) -> List[Participant]:
|
||||
primary = [item for item in participants if item.role in {"host", "speaker"}]
|
||||
if not primary:
|
||||
primary = participants
|
||||
|
||||
if not timeline or not timeline.participant_first_seen:
|
||||
return primary[:10]
|
||||
|
||||
active = set(timeline.active_speakers_at(seconds))
|
||||
visible = []
|
||||
for participant in primary:
|
||||
first_seen = timeline.first_seen_at(participant.username)
|
||||
if participant.role == "host" or first_seen is None and participant.username.lower() in active:
|
||||
first_seen = 0
|
||||
if first_seen is not None and first_seen <= seconds:
|
||||
visible.append((first_seen, participant))
|
||||
|
||||
visible.sort(key=lambda item: (item[0], item[1].username.lower()))
|
||||
selected = [item[1] for item in visible[-10:]]
|
||||
for active_username in active:
|
||||
if all(item.username.lower() != active_username for item in selected):
|
||||
active_participant = next((item for item in primary if item.username.lower() == active_username), None)
|
||||
if active_participant:
|
||||
selected = (selected + [active_participant])[-10:]
|
||||
return selected
|
||||
|
||||
|
||||
def participant_layout(width: int, height: int, count: int) -> List[Tuple[int, int, int, int]]:
|
||||
top = 142
|
||||
bottom = height - 28
|
||||
area_width = width - 88
|
||||
area_height = bottom - top
|
||||
|
||||
columns = 1 if count == 1 else 2
|
||||
rows = int(math.ceil(count / columns))
|
||||
cell_width = area_width // columns
|
||||
cell_height = area_height // rows
|
||||
|
||||
boxes = []
|
||||
for index in range(count):
|
||||
col = index % columns
|
||||
row = index // columns
|
||||
x1 = 44 + col * cell_width
|
||||
y1 = top + row * cell_height
|
||||
x2 = x1 + cell_width - 18
|
||||
y2 = y1 + cell_height - 10
|
||||
boxes.append((x1, y1, x2, y2))
|
||||
return boxes
|
||||
|
||||
|
||||
def draw_participant(
|
||||
frame: Image.Image,
|
||||
draw: ImageDraw.ImageDraw,
|
||||
participant: Participant,
|
||||
avatar: Optional[Image.Image],
|
||||
box: Tuple[int, int, int, int],
|
||||
active: Set[str],
|
||||
seconds: float,
|
||||
fonts: Dict[str, ImageFont.ImageFont],
|
||||
) -> None:
|
||||
x1, y1, x2, y2 = box
|
||||
is_active = participant.username.lower() in active
|
||||
fill = (25, 30, 35) if not is_active else (20, 38, 34)
|
||||
outline = (52, 63, 73) if not is_active else (29, 155, 120)
|
||||
draw_rounded_rectangle(draw, box, 20, fill=fill, outline=outline, width=3 if is_active else 1)
|
||||
|
||||
cell_height = y2 - y1
|
||||
avatar_size = min(118, max(44, int(cell_height * 0.42)))
|
||||
avatar_x = x1 + ((x2 - x1) - avatar_size) // 2
|
||||
avatar_y = y1 + max(8, int(cell_height * 0.08))
|
||||
if avatar is None:
|
||||
avatar = placeholder_avatar(participant, avatar_size)
|
||||
resized_avatar = avatar.resize((avatar_size, avatar_size))
|
||||
frame.paste(resized_avatar, (avatar_x, avatar_y), resized_avatar.getchannel("A"))
|
||||
|
||||
if is_active:
|
||||
pulse = 7
|
||||
draw.ellipse(
|
||||
(
|
||||
avatar_x - pulse,
|
||||
avatar_y - pulse,
|
||||
avatar_x + avatar_size + pulse,
|
||||
avatar_y + avatar_size + pulse,
|
||||
),
|
||||
outline=(29, 155, 120),
|
||||
width=4,
|
||||
)
|
||||
|
||||
text_y = avatar_y + avatar_size + 6
|
||||
text_width = max(24, (x2 - x1) - 22)
|
||||
name = ellipsize(draw, participant.display_name, fonts["name"], text_width)
|
||||
username = ellipsize(draw, f"@{participant.username}", fonts["small"], text_width)
|
||||
draw_centered_text(draw, name, (x1 + 10, text_y, x2 - 10, text_y + 22), fonts["name"], (238, 242, 246))
|
||||
draw_centered_text(draw, username, (x1 + 10, text_y + 22, x2 - 10, text_y + 40), fonts["small"], (139, 152, 165))
|
||||
|
||||
if is_active:
|
||||
badge = "Speaking"
|
||||
text_width = draw.textlength(badge, font=fonts["small"])
|
||||
badge_box = (x1 + 12, y1 + 10, x1 + int(text_width) + 34, y1 + 36)
|
||||
draw_rounded_rectangle(draw, badge_box, 14, fill=(29, 155, 120))
|
||||
draw.text((badge_box[0] + 13, badge_box[1] + 6), badge, fill=(3, 18, 14), font=fonts["small"])
|
||||
|
||||
|
||||
def placeholder_avatar(participant: Participant, size: int) -> Image.Image:
|
||||
colors = [(29, 155, 240), (249, 24, 128), (255, 212, 0), (120, 86, 255), (0, 186, 124)]
|
||||
index = sum(ord(char) for char in participant.username) % len(colors)
|
||||
image = Image.new("RGB", (size, size), colors[index])
|
||||
draw = ImageDraw.Draw(image)
|
||||
font = load_fonts()["initial"]
|
||||
initial = (participant.display_name or participant.username or "?")[0].upper()
|
||||
bbox = draw.textbbox((0, 0), initial, font=font)
|
||||
draw.text(((size - (bbox[2] - bbox[0])) / 2, (size - (bbox[3] - bbox[1])) / 2 - 6), initial, fill=(255, 255, 255), font=font)
|
||||
return crop_circle(image, size)
|
||||
|
||||
|
||||
def crop_circle(image: Image.Image, size: int) -> Image.Image:
|
||||
source = image.convert("RGB").resize((size, size))
|
||||
mask = Image.new("L", (size, size), 0)
|
||||
draw = ImageDraw.Draw(mask)
|
||||
draw.ellipse((0, 0, size - 1, size - 1), fill=255)
|
||||
output = Image.new("RGBA", (size, size), (0, 0, 0, 0))
|
||||
output.paste(source, (0, 0), mask)
|
||||
return output
|
||||
|
||||
|
||||
def load_fonts() -> Dict[str, ImageFont.ImageFont]:
|
||||
candidates = [
|
||||
"/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
|
||||
"/usr/share/fonts/truetype/liberation2/LiberationSans-Regular.ttf",
|
||||
]
|
||||
bold_candidates = [
|
||||
"/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf",
|
||||
"/usr/share/fonts/truetype/liberation2/LiberationSans-Bold.ttf",
|
||||
]
|
||||
regular = next((path for path in candidates if os.path.exists(path)), None)
|
||||
bold = next((path for path in bold_candidates if os.path.exists(path)), regular)
|
||||
return {
|
||||
"title": ImageFont.truetype(bold, 28) if bold else ImageFont.load_default(),
|
||||
"name": ImageFont.truetype(bold, 16) if bold else ImageFont.load_default(),
|
||||
"small": ImageFont.truetype(regular, 13) if regular else ImageFont.load_default(),
|
||||
"initial": ImageFont.truetype(bold, 64) if bold else ImageFont.load_default(),
|
||||
}
|
||||
|
||||
|
||||
def draw_centered_text(draw: ImageDraw.ImageDraw, text: str, box: Tuple[int, int, int, int], font, fill) -> None:
|
||||
bbox = draw.textbbox((0, 0), text, font=font)
|
||||
x = box[0] + ((box[2] - box[0]) - (bbox[2] - bbox[0])) / 2
|
||||
y = box[1] + ((box[3] - box[1]) - (bbox[3] - bbox[1])) / 2
|
||||
draw.text((x, y), text, fill=fill, font=font)
|
||||
|
||||
|
||||
def draw_rounded_rectangle(draw: ImageDraw.ImageDraw, box, radius: int, fill, outline=None, width: int = 1) -> None:
|
||||
draw.rounded_rectangle(box, radius=radius, fill=fill, outline=outline, width=width)
|
||||
|
||||
|
||||
def truncate(value: str, max_length: int) -> str:
|
||||
if len(value) <= max_length:
|
||||
return value
|
||||
return value[: max_length - 1] + "..."
|
||||
|
||||
|
||||
def ellipsize(draw: ImageDraw.ImageDraw, value: str, font, max_width: int) -> str:
|
||||
if draw.textlength(value, font=font) <= max_width:
|
||||
return value
|
||||
suffix = "..."
|
||||
trimmed = value
|
||||
while trimmed and draw.textlength(trimmed + suffix, font=font) > max_width:
|
||||
trimmed = trimmed[:-1]
|
||||
return (trimmed or value[:1]) + suffix
|
||||
|
||||
|
||||
def format_time(seconds: float) -> str:
|
||||
minutes, secs = divmod(int(seconds), 60)
|
||||
hours, minutes = divmod(minutes, 60)
|
||||
if hours:
|
||||
return f"{hours:d}:{minutes:02d}:{secs:02d}"
|
||||
return f"{minutes:d}:{secs:02d}"
|
||||
478
src/x_api.py
Normal file
478
src/x_api.py
Normal file
@@ -0,0 +1,478 @@
|
||||
import json
|
||||
import os
|
||||
from dataclasses import replace
|
||||
from datetime import datetime
|
||||
from http.cookiejar import MozillaCookieJar
|
||||
from typing import Dict, Iterable, List, Optional, Tuple
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import requests
|
||||
|
||||
from src.models import Participant, SpaceMetadata, SpeakerInterval
|
||||
|
||||
|
||||
DEFAULT_GRAPHQL_QUERY_IDS = (
|
||||
"DovcWBliWe-v2TCRGFtibA",
|
||||
"Uv5R_-Chxbn1FEkyUkSW2w",
|
||||
"FJoTSHMVF7fMhGLc2t9cog",
|
||||
)
|
||||
|
||||
GRAPHQL_FEATURES = {
|
||||
"spaces_2022_h2_spaces_communities": True,
|
||||
"spaces_2022_h2_clipping": True,
|
||||
"creator_subscriptions_tweet_preview_api_enabled": True,
|
||||
"responsive_web_graphql_skip_user_profile_image_extensions_enabled": False,
|
||||
}
|
||||
|
||||
|
||||
class XApiError(Exception):
|
||||
"""Raised when X metadata cannot be fetched or parsed."""
|
||||
|
||||
|
||||
def load_netscape_cookies(cookies_file: str) -> Dict[str, str]:
|
||||
"""Load a Netscape cookies.txt file into a simple name/value mapping."""
|
||||
jar = MozillaCookieJar()
|
||||
jar.load(cookies_file, ignore_discard=True, ignore_expires=True)
|
||||
return {cookie.name: cookie.value for cookie in jar}
|
||||
|
||||
|
||||
def cookie_header(cookies: Dict[str, str]) -> str:
|
||||
return "; ".join(f"{name}={value}" for name, value in cookies.items())
|
||||
|
||||
|
||||
class XApiClient:
|
||||
"""Minimal client for X web GraphQL metadata used by Spaces replays."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cookies_file: Optional[str] = None,
|
||||
bearer_token: Optional[str] = None,
|
||||
query_ids: Iterable[str] = DEFAULT_GRAPHQL_QUERY_IDS,
|
||||
timeout: int = 20,
|
||||
):
|
||||
self.cookies = load_netscape_cookies(cookies_file) if cookies_file else {}
|
||||
self.bearer_token = bearer_token or os.getenv("X_BEARER_TOKEN")
|
||||
if not self.bearer_token:
|
||||
raise XApiError(
|
||||
"No X API bearer token available. Set the X_BEARER_TOKEN environment "
|
||||
"variable to a bearer token from x.com's web client (visible in browser "
|
||||
"devtools as the 'authorization: Bearer ...' header on requests to "
|
||||
"x.com/i/api)."
|
||||
)
|
||||
env_query_id = os.getenv("X_AUDIO_SPACE_QUERY_ID")
|
||||
self.query_ids = (env_query_id, *tuple(query_ids)) if env_query_id else tuple(query_ids)
|
||||
self.timeout = timeout
|
||||
self.session = requests.Session()
|
||||
|
||||
def fetch_space_metadata(self, space_id: str) -> SpaceMetadata:
|
||||
last_error: Optional[Exception] = None
|
||||
for query_id in self.query_ids:
|
||||
try:
|
||||
payload = self._get_audio_space_by_id(query_id, space_id)
|
||||
return parse_audio_space_metadata(payload, space_id)
|
||||
except Exception as exc:
|
||||
last_error = exc
|
||||
|
||||
raise XApiError(f"Could not fetch Space metadata: {last_error}")
|
||||
|
||||
def fetch_space_timeline(self, metadata: SpaceMetadata) -> Tuple[List[SpeakerInterval], Dict[str, float], List[Participant]]:
|
||||
if not metadata.media_key:
|
||||
raise XApiError("Space metadata does not include media_key")
|
||||
|
||||
status = self.fetch_live_video_status(metadata.media_key)
|
||||
playlist_url = (status.get("source") or {}).get("location")
|
||||
start_ms = fetch_playlist_start_ms(playlist_url, self.session) if playlist_url else None
|
||||
if start_ms is None:
|
||||
start_ms = metadata.started_at_ms
|
||||
if start_ms is None:
|
||||
raise XApiError("Could not determine replay start time")
|
||||
|
||||
end_ms = metadata.ended_at_ms
|
||||
if end_ms is None:
|
||||
end_ms = start_ms + int((status.get("duration") or 4 * 60 * 60) * 1000)
|
||||
|
||||
access = self.access_chat(status.get("chatToken"))
|
||||
endpoint = access.get("replay_endpoint") or access.get("endpoint")
|
||||
access_token = access.get("replay_access_token") or access.get("access_token")
|
||||
if not endpoint or not access_token:
|
||||
raise XApiError("Chat access response did not include replay endpoint/token")
|
||||
|
||||
messages = self.fetch_history_messages(endpoint, access_token, start_ms, end_ms)
|
||||
return build_timeline_from_history(messages, start_ms)
|
||||
|
||||
def fetch_live_video_status(self, media_key: str) -> dict:
|
||||
url = f"https://x.com/i/api/1.1/live_video_stream/status/{media_key}"
|
||||
headers = self._x_headers()
|
||||
response = self.session.get(
|
||||
url,
|
||||
params={
|
||||
"client": "web",
|
||||
"use_syndication_guest_id": "false",
|
||||
"cookie_set_host": "x.com",
|
||||
},
|
||||
headers=headers,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
if response.status_code >= 400:
|
||||
raise XApiError(f"live_video_stream/status returned HTTP {response.status_code}")
|
||||
return response.json()
|
||||
|
||||
def access_chat(self, chat_token: Optional[str]) -> dict:
|
||||
if not chat_token:
|
||||
raise XApiError("live_video_stream/status did not include chatToken")
|
||||
|
||||
payload = {"chat_token": chat_token}
|
||||
urls = (
|
||||
"https://proxsee-cf.pscp.tv/api/v2/accessChat",
|
||||
"https://proxsee.pscp.tv/api/v2/accessChat",
|
||||
"https://proxsee.pscp.tv/api/v2/accessChatPublic",
|
||||
)
|
||||
last_error = None
|
||||
for url in urls:
|
||||
try:
|
||||
response = self.session.post(url, json=payload, headers=chat_headers(), timeout=self.timeout)
|
||||
if response.status_code < 400 and response.text:
|
||||
return response.json()
|
||||
last_error = XApiError(f"{url} returned HTTP {response.status_code}")
|
||||
except Exception as exc:
|
||||
last_error = exc
|
||||
raise XApiError(f"Could not access chat replay: {last_error}")
|
||||
|
||||
def fetch_history_messages(self, endpoint: str, access_token: str, start_ms: int, end_ms: int) -> List[dict]:
|
||||
url = endpoint.rstrip("/") + "/chatapi/v1/history"
|
||||
step_ns = 60 * 1_000_000_000
|
||||
cursor_ns = start_ms * 1_000_000
|
||||
end_ns = end_ms * 1_000_000
|
||||
seen_cursors = set()
|
||||
seen_messages = set()
|
||||
messages: List[dict] = []
|
||||
|
||||
first_payload = self._history_page(url, access_token, "", None)
|
||||
messages.extend(unique_messages(first_payload.get("messages") or [], seen_messages))
|
||||
response_cursor = _to_int(first_payload.get("cursor"))
|
||||
if response_cursor and response_cursor > cursor_ns:
|
||||
cursor_ns = response_cursor
|
||||
|
||||
while cursor_ns <= end_ns:
|
||||
if cursor_ns in seen_cursors:
|
||||
cursor_ns += step_ns
|
||||
continue
|
||||
seen_cursors.add(cursor_ns)
|
||||
payload = self._history_page(url, access_token, str(cursor_ns), None)
|
||||
page_messages = unique_messages(payload.get("messages") or [], seen_messages)
|
||||
messages.extend(page_messages)
|
||||
response_cursor = _to_int(payload.get("cursor"))
|
||||
if response_cursor and response_cursor > cursor_ns:
|
||||
cursor_ns = response_cursor
|
||||
else:
|
||||
cursor_ns += step_ns
|
||||
|
||||
return messages
|
||||
|
||||
def _history_page(self, url: str, access_token: str, cursor: str, since: Optional[int]) -> dict:
|
||||
response = self.session.post(
|
||||
url,
|
||||
json={
|
||||
"access_token": access_token,
|
||||
"cursor": cursor,
|
||||
"limit": 1000,
|
||||
"since": since,
|
||||
"quick_get": True,
|
||||
},
|
||||
headers=chat_headers(),
|
||||
timeout=self.timeout,
|
||||
)
|
||||
if response.status_code >= 400:
|
||||
raise XApiError(f"history returned HTTP {response.status_code}")
|
||||
return response.json()
|
||||
|
||||
def _x_headers(self) -> Dict[str, str]:
|
||||
headers = {
|
||||
"accept": "*/*",
|
||||
"authorization": f"Bearer {self.bearer_token}",
|
||||
"content-type": "application/json",
|
||||
"referer": "https://x.com/",
|
||||
"user-agent": (
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"
|
||||
),
|
||||
"x-twitter-active-user": "yes",
|
||||
"x-twitter-auth-type": "OAuth2Session",
|
||||
"x-twitter-client-language": "en",
|
||||
}
|
||||
if self.cookies:
|
||||
headers["cookie"] = cookie_header(self.cookies)
|
||||
if self.cookies.get("ct0"):
|
||||
headers["x-csrf-token"] = self.cookies["ct0"]
|
||||
return headers
|
||||
|
||||
def _get_audio_space_by_id(self, query_id: str, space_id: str) -> dict:
|
||||
variables = {
|
||||
"id": space_id,
|
||||
"isMetatagsQuery": False,
|
||||
"withReplays": True,
|
||||
"withListeners": True,
|
||||
}
|
||||
params = urlencode(
|
||||
{
|
||||
"variables": json.dumps(variables, separators=(",", ":")),
|
||||
"features": json.dumps(GRAPHQL_FEATURES, separators=(",", ":")),
|
||||
}
|
||||
)
|
||||
url = f"https://x.com/i/api/graphql/{query_id}/AudioSpaceById?{params}"
|
||||
headers = self._x_headers()
|
||||
|
||||
response = self.session.get(url, headers=headers, timeout=self.timeout)
|
||||
if response.status_code >= 400:
|
||||
raise XApiError(f"X returned HTTP {response.status_code}")
|
||||
|
||||
payload = response.json()
|
||||
if payload.get("errors"):
|
||||
raise XApiError(payload["errors"][0].get("message", "GraphQL error"))
|
||||
return payload
|
||||
|
||||
|
||||
def parse_audio_space_metadata(payload: dict, space_id: str) -> SpaceMetadata:
|
||||
audio_space = payload.get("data", {}).get("audioSpace") or {}
|
||||
metadata = audio_space.get("metadata") or {}
|
||||
if not metadata:
|
||||
raise XApiError("AudioSpaceById response did not include metadata")
|
||||
|
||||
title = metadata.get("title") or f"X Space {space_id}"
|
||||
participants = _extract_participants(audio_space)
|
||||
creator = _participant_from_user_result(metadata.get("creator_results"), "host")
|
||||
if creator:
|
||||
participants = _dedupe_participants([creator, *participants])
|
||||
|
||||
return SpaceMetadata(
|
||||
space_id=metadata.get("rest_id") or space_id,
|
||||
title=title,
|
||||
participants=participants,
|
||||
media_key=metadata.get("media_key"),
|
||||
started_at_ms=_timestamp_to_ms(metadata.get("started_at") or metadata.get("scheduled_start")),
|
||||
ended_at_ms=_timestamp_to_ms(metadata.get("ended_at")),
|
||||
)
|
||||
|
||||
|
||||
def chat_headers() -> Dict[str, str]:
|
||||
return {
|
||||
"accept": "*/*",
|
||||
"content-type": "application/json",
|
||||
"origin": "https://x.com",
|
||||
"referer": "https://x.com/",
|
||||
"user-agent": (
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"
|
||||
),
|
||||
"x-periscope-user-agent": "Twitter/m5",
|
||||
}
|
||||
|
||||
|
||||
def fetch_playlist_start_ms(playlist_url: str, session: requests.Session) -> Optional[int]:
|
||||
response = session.get(playlist_url, headers={"origin": "https://x.com", "referer": "https://x.com/"}, timeout=20)
|
||||
if response.status_code >= 400:
|
||||
return None
|
||||
for line in response.text.splitlines():
|
||||
if line.startswith("#EXT-X-PROGRAM-DATE-TIME:"):
|
||||
return iso_to_ms(line.split(":", 1)[1].strip())
|
||||
return None
|
||||
|
||||
|
||||
def build_timeline_from_history(messages: List[dict], start_ms: int) -> Tuple[List[SpeakerInterval], Dict[str, float], List[Participant]]:
|
||||
participant_first_seen: Dict[str, float] = {}
|
||||
participants: Dict[str, Participant] = {}
|
||||
open_microphones: Dict[str, float] = {}
|
||||
intervals: List[SpeakerInterval] = []
|
||||
|
||||
for message in messages:
|
||||
decoded = decode_history_payload(message)
|
||||
if not decoded:
|
||||
continue
|
||||
|
||||
body = decoded.get("body")
|
||||
if isinstance(body, str):
|
||||
try:
|
||||
body = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
body = None
|
||||
if not isinstance(body, dict):
|
||||
continue
|
||||
|
||||
body_type = body.get("type")
|
||||
if body_type == 40:
|
||||
username = body.get("guestUsername")
|
||||
event_ms = ns_to_ms(decoded.get("timestamp"))
|
||||
if username and event_ms:
|
||||
username_key = username.lower()
|
||||
seconds = max(0.0, (event_ms - start_ms) / 1000)
|
||||
event = body.get("guestBroadcastingEvent")
|
||||
if event in {9, 17}:
|
||||
participant_first_seen.setdefault(username_key, seconds)
|
||||
participants.setdefault(
|
||||
username_key,
|
||||
Participant(
|
||||
user_id=str(body.get("guestRemoteID") or username),
|
||||
display_name=str(username),
|
||||
username=str(username),
|
||||
role="speaker",
|
||||
),
|
||||
)
|
||||
if event == 17:
|
||||
open_microphones[username_key] = seconds
|
||||
elif event == 16:
|
||||
start = open_microphones.pop(username_key, None)
|
||||
if start is not None and seconds > start:
|
||||
intervals.append(SpeakerInterval(start=start, end=seconds, speaker=username_key))
|
||||
elif body_type == 45:
|
||||
username = body.get("username")
|
||||
event_ms = iso_to_ms(body.get("programDateTime")) or _to_int(body.get("timestamp"))
|
||||
if username and event_ms:
|
||||
participants.setdefault(
|
||||
username.lower(),
|
||||
Participant(
|
||||
user_id=str(body.get("twitter_id") or body.get("user_id") or username),
|
||||
display_name=str(body.get("displayName") or username),
|
||||
username=str(username),
|
||||
role="speaker",
|
||||
),
|
||||
)
|
||||
|
||||
last_event_second = max((interval.end for interval in intervals), default=0)
|
||||
for username, start in open_microphones.items():
|
||||
if last_event_second > start:
|
||||
intervals.append(SpeakerInterval(start=start, end=last_event_second, speaker=username))
|
||||
|
||||
return intervals, participant_first_seen, list(participants.values())
|
||||
|
||||
|
||||
def unique_messages(messages: List[dict], seen: set) -> List[dict]:
|
||||
output = []
|
||||
for message in messages:
|
||||
key = message.get("signature") or message.get("payload")
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
output.append(message)
|
||||
return output
|
||||
|
||||
|
||||
def decode_history_payload(message: dict) -> Optional[dict]:
|
||||
payload = message.get("payload")
|
||||
if not isinstance(payload, str):
|
||||
return None
|
||||
try:
|
||||
return json.loads(payload)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
|
||||
def iso_to_ms(value: Optional[str]) -> Optional[int]:
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
normalized = value.replace("Z", "+00:00")
|
||||
return int(datetime.fromisoformat(normalized).timestamp() * 1000)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def ns_to_ms(value) -> Optional[int]:
|
||||
number = _to_int(value)
|
||||
if number is None:
|
||||
return None
|
||||
return number // 1_000_000
|
||||
|
||||
|
||||
def _extract_participants(audio_space: dict) -> List[Participant]:
|
||||
participants_section = audio_space.get("participants") or {}
|
||||
participants: List[Participant] = []
|
||||
|
||||
for key, role in (("admins", "host"), ("speakers", "speaker"), ("listeners", "listener")):
|
||||
for item in participants_section.get(key) or []:
|
||||
participant = _participant_from_participant_item(item, role)
|
||||
if participant:
|
||||
participants.append(participant)
|
||||
|
||||
return _dedupe_participants(participants)
|
||||
|
||||
|
||||
def _participant_from_participant_item(item: dict, role: str) -> Optional[Participant]:
|
||||
username = item.get("twitter_screen_name")
|
||||
display_name = item.get("display_name") or username
|
||||
avatar_url = item.get("avatar_url")
|
||||
|
||||
user_results = item.get("user_results") or {}
|
||||
user_id = user_results.get("rest_id") or item.get("periscope_user_id") or username
|
||||
user = _participant_from_user_result(user_results, role)
|
||||
if user:
|
||||
return replace(
|
||||
user,
|
||||
display_name=display_name or user.display_name,
|
||||
username=username or user.username,
|
||||
avatar_url=avatar_url or user.avatar_url,
|
||||
role=role,
|
||||
)
|
||||
|
||||
if not user_id or not username:
|
||||
return None
|
||||
return Participant(
|
||||
user_id=str(user_id),
|
||||
display_name=str(display_name or username),
|
||||
username=str(username),
|
||||
avatar_url=avatar_url,
|
||||
role=role,
|
||||
)
|
||||
|
||||
|
||||
def _participant_from_user_result(user_results: Optional[dict], role: str) -> Optional[Participant]:
|
||||
if not user_results:
|
||||
return None
|
||||
|
||||
result = user_results.get("result") or user_results
|
||||
legacy = result.get("legacy") or {}
|
||||
core = result.get("core") or {}
|
||||
avatar = result.get("avatar") or {}
|
||||
|
||||
username = legacy.get("screen_name") or core.get("screen_name")
|
||||
display_name = legacy.get("name") or core.get("name") or username
|
||||
avatar_url = legacy.get("profile_image_url_https") or avatar.get("image_url")
|
||||
user_id = result.get("rest_id") or user_results.get("rest_id") or result.get("id")
|
||||
|
||||
if not user_id or not username:
|
||||
return None
|
||||
return Participant(
|
||||
user_id=str(user_id),
|
||||
display_name=str(display_name or username),
|
||||
username=str(username),
|
||||
avatar_url=avatar_url,
|
||||
role=role,
|
||||
)
|
||||
|
||||
|
||||
def _dedupe_participants(participants: List[Participant]) -> List[Participant]:
|
||||
by_key: Dict[str, Participant] = {}
|
||||
role_rank = {"host": 3, "speaker": 2, "listener": 1}
|
||||
for participant in participants:
|
||||
key = participant.username.lower() or participant.user_id
|
||||
existing = by_key.get(key)
|
||||
if not existing or role_rank.get(participant.role, 0) > role_rank.get(existing.role, 0):
|
||||
by_key[key] = participant
|
||||
return list(by_key.values())
|
||||
|
||||
|
||||
def _to_int(value) -> Optional[int]:
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _timestamp_to_ms(value) -> Optional[int]:
|
||||
number = _to_int(value)
|
||||
if number is None:
|
||||
return None
|
||||
if number > 10_000_000_000:
|
||||
return number
|
||||
return number * 1000
|
||||
0
tests/__init__.py
Normal file
0
tests/__init__.py
Normal file
104
tests/test_downloader.py
Normal file
104
tests/test_downloader.py
Normal file
@@ -0,0 +1,104 @@
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from src.downloader import download_space, DownloadError
|
||||
|
||||
|
||||
def test_download_space_basic(tmp_path):
|
||||
"""Test basic download functionality."""
|
||||
space_url = "https://x.com/i/spaces/1ZkKzZWnVRRKv"
|
||||
|
||||
with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class:
|
||||
# Mock the YoutubeDL instance
|
||||
mock_ytdl = Mock()
|
||||
mock_ytdl_class.return_value.__enter__.return_value = mock_ytdl
|
||||
mock_ytdl.extract_info.return_value = {'id': '1ZkKzZWnVRRKv'}
|
||||
|
||||
# Call download_space
|
||||
result = download_space(
|
||||
space_url=space_url,
|
||||
output_dir=str(tmp_path),
|
||||
cookies_file=None
|
||||
)
|
||||
|
||||
# Verify YoutubeDL was called
|
||||
mock_ytdl.extract_info.assert_called_once_with(space_url, download=True)
|
||||
|
||||
# Verify result contains expected filename
|
||||
assert "space_1ZkKzZWnVRRKv" in result
|
||||
|
||||
|
||||
def test_download_space_with_cookies(tmp_path):
|
||||
"""Test that cookies file is passed to yt-dlp when provided."""
|
||||
space_url = "https://x.com/i/spaces/1ZkKzZWnVRRKv"
|
||||
cookies_path = "/path/to/cookies.txt"
|
||||
|
||||
with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class:
|
||||
# Mock the YoutubeDL instance
|
||||
mock_ytdl = Mock()
|
||||
mock_ytdl_class.return_value.__enter__.return_value = mock_ytdl
|
||||
mock_ytdl.extract_info.return_value = {'id': '1ZkKzZWnVRRKv'}
|
||||
|
||||
# Call download_space with cookies
|
||||
result = download_space(
|
||||
space_url=space_url,
|
||||
output_dir=str(tmp_path),
|
||||
cookies_file=cookies_path
|
||||
)
|
||||
|
||||
# Verify YoutubeDL was initialized with correct options
|
||||
call_args = mock_ytdl_class.call_args[0][0]
|
||||
assert 'cookiefile' in call_args
|
||||
assert call_args['cookiefile'] == cookies_path
|
||||
|
||||
|
||||
def test_download_space_ytdlp_configuration(tmp_path):
|
||||
"""Test that yt-dlp is configured correctly with format, outtmpl, and postprocessors."""
|
||||
space_url = "https://x.com/i/spaces/1ZkKzZWnVRRKv"
|
||||
|
||||
with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class:
|
||||
# Mock the YoutubeDL instance
|
||||
mock_ytdl = Mock()
|
||||
mock_ytdl_class.return_value.__enter__.return_value = mock_ytdl
|
||||
mock_ytdl.extract_info.return_value = {'id': '1ZkKzZWnVRRKv'}
|
||||
|
||||
# Call download_space
|
||||
result = download_space(
|
||||
space_url=space_url,
|
||||
output_dir=str(tmp_path),
|
||||
cookies_file=None
|
||||
)
|
||||
|
||||
# Verify YoutubeDL was initialized with correct options
|
||||
call_args = mock_ytdl_class.call_args[0][0]
|
||||
|
||||
# Check format
|
||||
assert call_args['format'] == 'bestaudio/best'
|
||||
|
||||
# Check outtmpl contains the space_id
|
||||
assert 'outtmpl' in call_args
|
||||
assert 'space_1ZkKzZWnVRRKv' in call_args['outtmpl']
|
||||
|
||||
# Check postprocessors for MP3 conversion
|
||||
assert 'postprocessors' in call_args
|
||||
assert len(call_args['postprocessors']) > 0
|
||||
assert call_args['postprocessors'][0]['key'] == 'FFmpegExtractAudio'
|
||||
assert call_args['postprocessors'][0]['preferredcodec'] == 'mp3'
|
||||
|
||||
|
||||
def test_download_space_error_handling(tmp_path):
|
||||
"""Test that download errors are properly caught and wrapped."""
|
||||
space_url = "https://x.com/i/spaces/invalid"
|
||||
|
||||
with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class:
|
||||
mock_ytdl = Mock()
|
||||
mock_ytdl_class.return_value.__enter__.return_value = mock_ytdl
|
||||
mock_ytdl.extract_info.side_effect = Exception("Video unavailable")
|
||||
|
||||
with pytest.raises(DownloadError) as exc_info:
|
||||
download_space(
|
||||
space_url=space_url,
|
||||
output_dir=str(tmp_path),
|
||||
cookies_file=None
|
||||
)
|
||||
|
||||
assert "Failed to download" in str(exc_info.value)
|
||||
33
tests/test_integration.py
Normal file
33
tests/test_integration.py
Normal file
@@ -0,0 +1,33 @@
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
from src.main import cli
|
||||
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_full_workflow_with_mocks(tmp_path):
|
||||
"""Integration test for complete workflow."""
|
||||
runner = CliRunner()
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
with patch('src.downloader.yt_dlp.YoutubeDL') as mock_ytdl_class:
|
||||
# Mock successful download
|
||||
mock_ytdl = mock_ytdl_class.return_value.__enter__.return_value
|
||||
mock_ytdl.extract_info.return_value = {'id': '1ZkKzZWnVRRKv'}
|
||||
|
||||
# Run CLI
|
||||
result = runner.invoke(cli, [
|
||||
'https://x.com/i/spaces/1ZkKzZWnVRRKv',
|
||||
'--output', str(tmp_path),
|
||||
'--mode', 'audio',
|
||||
])
|
||||
|
||||
# Verify success
|
||||
assert result.exit_code == 0
|
||||
assert "Successfully downloaded" in result.output
|
||||
|
||||
# Verify yt-dlp was called correctly
|
||||
mock_ytdl.extract_info.assert_called_once()
|
||||
call_args = mock_ytdl_class.call_args
|
||||
assert 'bestaudio' in call_args[0][0]['format']
|
||||
assert 'mp3' in call_args[0][0]['postprocessors'][0]['preferredcodec']
|
||||
100
tests/test_main.py
Normal file
100
tests/test_main.py
Normal file
@@ -0,0 +1,100 @@
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
from unittest.mock import patch, Mock
|
||||
from src.main import cli
|
||||
from src.downloader import DownloadError
|
||||
|
||||
|
||||
def test_cli_valid_url(tmp_path):
|
||||
"""Test CLI with valid URL."""
|
||||
runner = CliRunner()
|
||||
|
||||
with patch('src.main.validate_space_url') as mock_validate, \
|
||||
patch('src.main.download_space') as mock_download:
|
||||
|
||||
mock_validate.return_value = (True, "1ZkKzZWnVRRKv", None)
|
||||
mock_download.return_value = str(tmp_path / "space_1ZkKzZWnVRRKv.mp3")
|
||||
|
||||
result = runner.invoke(cli, ['https://x.com/i/spaces/1ZkKzZWnVRRKv', '--mode', 'audio'])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "Successfully downloaded" in result.output
|
||||
assert "space_1ZkKzZWnVRRKv.mp3" in result.output
|
||||
|
||||
|
||||
def test_cli_invalid_url():
|
||||
"""Test CLI with invalid URL."""
|
||||
runner = CliRunner()
|
||||
|
||||
result = runner.invoke(cli, ['https://invalid.com/spaces/123'])
|
||||
|
||||
assert result.exit_code == 1
|
||||
assert "Invalid URL format" in result.output
|
||||
|
||||
|
||||
def test_cli_download_failure():
|
||||
"""Test CLI when download fails."""
|
||||
runner = CliRunner()
|
||||
|
||||
with patch('src.main.validate_space_url') as mock_validate, \
|
||||
patch('src.main.download_space') as mock_download:
|
||||
|
||||
mock_validate.return_value = (True, "1ZkKzZWnVRRKv", None)
|
||||
mock_download.side_effect = DownloadError("Video unavailable")
|
||||
|
||||
result = runner.invoke(cli, ['https://x.com/i/spaces/1ZkKzZWnVRRKv', '--mode', 'audio'])
|
||||
|
||||
assert result.exit_code == 2
|
||||
assert "Video unavailable" in result.output
|
||||
|
||||
|
||||
def test_cli_with_cookies(tmp_path):
|
||||
"""Test CLI with cookies file."""
|
||||
runner = CliRunner()
|
||||
|
||||
# Create a fake cookies file
|
||||
cookies_file = tmp_path / "cookies.txt"
|
||||
cookies_file.write_text("# Netscape HTTP Cookie File\n")
|
||||
|
||||
with patch('src.main.validate_space_url') as mock_validate, \
|
||||
patch('src.main.download_space') as mock_download:
|
||||
|
||||
mock_validate.return_value = (True, "1ZkKzZWnVRRKv", None)
|
||||
mock_download.return_value = str(tmp_path / "space_1ZkKzZWnVRRKv.mp3")
|
||||
|
||||
result = runner.invoke(cli, [
|
||||
'https://x.com/i/spaces/1ZkKzZWnVRRKv',
|
||||
'--cookies', str(cookies_file),
|
||||
'--mode', 'audio',
|
||||
])
|
||||
|
||||
assert result.exit_code == 0
|
||||
# Verify the call was made with correct arguments including cookies_file
|
||||
mock_download.assert_called_once_with(
|
||||
'https://x.com/i/spaces/1ZkKzZWnVRRKv',
|
||||
'.',
|
||||
str(cookies_file)
|
||||
)
|
||||
|
||||
|
||||
def test_cli_video_mode(tmp_path):
|
||||
"""Test CLI default video mode."""
|
||||
runner = CliRunner()
|
||||
|
||||
with patch('src.main.validate_space_url') as mock_validate, \
|
||||
patch('src.main.download_space_video') as mock_download:
|
||||
|
||||
mock_validate.return_value = (True, "1ZkKzZWnVRRKv", None)
|
||||
mock_download.return_value = str(tmp_path / "space_1ZkKzZWnVRRKv.mp4")
|
||||
|
||||
result = runner.invoke(cli, ['https://x.com/i/spaces/1ZkKzZWnVRRKv'])
|
||||
|
||||
assert result.exit_code == 0
|
||||
assert "space_1ZkKzZWnVRRKv.mp4" in result.output
|
||||
mock_download.assert_called_once_with(
|
||||
'https://x.com/i/spaces/1ZkKzZWnVRRKv',
|
||||
'.',
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
)
|
||||
38
tests/test_validator.py
Normal file
38
tests/test_validator.py
Normal file
@@ -0,0 +1,38 @@
|
||||
import pytest
|
||||
from src.validator import validate_space_url
|
||||
|
||||
|
||||
def test_validate_space_url_valid():
|
||||
"""Test that valid X Space URL is accepted."""
|
||||
url = "https://x.com/i/spaces/1ZkKzZWnVRRKv"
|
||||
is_valid, space_id, error = validate_space_url(url)
|
||||
|
||||
assert is_valid is True
|
||||
assert space_id == "1ZkKzZWnVRRKv"
|
||||
assert error is None
|
||||
|
||||
|
||||
def test_validate_space_url_invalid_format():
|
||||
"""Test that invalid URLs are rejected."""
|
||||
invalid_urls = [
|
||||
"https://x.com/spaces/1ZkKzZWnVRRKv", # Missing /i/
|
||||
"https://x.com/i/spaces/", # Missing space ID
|
||||
"not a url", # Completely invalid
|
||||
"", # Empty string
|
||||
]
|
||||
|
||||
for url in invalid_urls:
|
||||
is_valid, space_id, error = validate_space_url(url)
|
||||
assert is_valid is False
|
||||
assert space_id is None
|
||||
assert "Invalid URL format" in error
|
||||
|
||||
|
||||
def test_validate_space_url_twitter_domain():
|
||||
"""Test that twitter.com URLs are also accepted."""
|
||||
url = "https://twitter.com/i/spaces/1ZkKzZWnVRRKv"
|
||||
is_valid, space_id, error = validate_space_url(url)
|
||||
|
||||
assert is_valid is True
|
||||
assert space_id == "1ZkKzZWnVRRKv"
|
||||
assert error is None
|
||||
47
tests/test_video.py
Normal file
47
tests/test_video.py
Normal file
@@ -0,0 +1,47 @@
|
||||
import json
|
||||
|
||||
from src.models import Participant, SpaceMetadata
|
||||
from src.video import SpeakerTimeline, build_ffmpeg_command, render_frame
|
||||
|
||||
|
||||
def test_speaker_timeline_reads_active_speaker(tmp_path):
|
||||
timeline_file = tmp_path / "timeline.json"
|
||||
timeline_file.write_text(
|
||||
json.dumps([
|
||||
{"start": 0, "end": 2.5, "speaker": "alice"},
|
||||
{"start": 2.5, "end": 4, "speaker": "@bob"},
|
||||
]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
timeline = SpeakerTimeline.from_file(str(timeline_file))
|
||||
|
||||
assert timeline.active_at(1) == "alice"
|
||||
assert timeline.active_at(3) == "bob"
|
||||
assert timeline.active_at(5) is None
|
||||
|
||||
|
||||
def test_build_ffmpeg_command_uses_audio_and_raw_video():
|
||||
command = build_ffmpeg_command("audio.mp3", "out.mp4", 720, 720, 24)
|
||||
|
||||
assert command[:2] == ["ffmpeg", "-y"]
|
||||
assert "audio.mp3" in command
|
||||
assert "out.mp4" in command
|
||||
assert "720x720" in command
|
||||
|
||||
|
||||
def test_render_frame_returns_rgb_image():
|
||||
metadata = SpaceMetadata(
|
||||
space_id="1ZkKzZWnVRRKv",
|
||||
title="Example Space",
|
||||
participants=[
|
||||
Participant("1", "Alice", "alice", role="host"),
|
||||
Participant("2", "Bob", "bob", role="speaker"),
|
||||
],
|
||||
)
|
||||
timeline = SpeakerTimeline.from_file_data([{"start": 0, "end": 10, "speaker": "bob"}])
|
||||
|
||||
frame = render_frame(metadata, {}, timeline, 1.0, 720, 720)
|
||||
|
||||
assert frame.mode == "RGB"
|
||||
assert frame.size == (720, 720)
|
||||
89
tests/test_x_api.py
Normal file
89
tests/test_x_api.py
Normal file
@@ -0,0 +1,89 @@
|
||||
import json
|
||||
|
||||
from src.x_api import build_timeline_from_history, parse_audio_space_metadata
|
||||
|
||||
|
||||
def test_parse_audio_space_metadata_extracts_title_creator_and_speakers():
|
||||
payload = {
|
||||
"data": {
|
||||
"audioSpace": {
|
||||
"metadata": {
|
||||
"rest_id": "1ZkKzZWnVRRKv",
|
||||
"title": "Example Space",
|
||||
"media_key": "28_123",
|
||||
"creator_results": {
|
||||
"result": {
|
||||
"rest_id": "10",
|
||||
"legacy": {
|
||||
"name": "Host User",
|
||||
"screen_name": "host",
|
||||
"profile_image_url_https": "https://example.com/host.jpg",
|
||||
},
|
||||
}
|
||||
},
|
||||
},
|
||||
"participants": {
|
||||
"speakers": [
|
||||
{
|
||||
"display_name": "Speaker User",
|
||||
"twitter_screen_name": "speaker",
|
||||
"avatar_url": "https://example.com/speaker.jpg",
|
||||
"periscope_user_id": "20",
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
metadata = parse_audio_space_metadata(payload, "1ZkKzZWnVRRKv")
|
||||
|
||||
assert metadata.title == "Example Space"
|
||||
assert metadata.media_key == "28_123"
|
||||
assert [participant.username for participant in metadata.participants] == ["host", "speaker"]
|
||||
assert metadata.participants[0].role == "host"
|
||||
assert metadata.participants[1].role == "speaker"
|
||||
|
||||
|
||||
def test_build_timeline_uses_unmute_mute_events_not_captions():
|
||||
start_ms = 1_000_000
|
||||
messages = [
|
||||
_history_message("alice", 17, start_ms + 5_000),
|
||||
_caption_message("bob", start_ms + 6_000),
|
||||
_history_message("alice", 16, start_ms + 12_000),
|
||||
]
|
||||
|
||||
intervals, first_seen, participants = build_timeline_from_history(messages, start_ms)
|
||||
|
||||
assert len(intervals) == 1
|
||||
assert intervals[0].speaker == "alice"
|
||||
assert intervals[0].start == 5
|
||||
assert intervals[0].end == 12
|
||||
assert first_seen == {"alice": 5}
|
||||
assert {participant.username for participant in participants} == {"alice", "bob"}
|
||||
|
||||
|
||||
def _history_message(username, event, timestamp_ms):
|
||||
payload = {
|
||||
"body": json.dumps({
|
||||
"guestBroadcastingEvent": event,
|
||||
"guestRemoteID": username,
|
||||
"guestUsername": username,
|
||||
"type": 40,
|
||||
}),
|
||||
"timestamp": timestamp_ms * 1_000_000,
|
||||
}
|
||||
return {"payload": json.dumps(payload)}
|
||||
|
||||
|
||||
def _caption_message(username, timestamp_ms):
|
||||
payload = {
|
||||
"body": json.dumps({
|
||||
"type": 45,
|
||||
"username": username,
|
||||
"displayName": username.title(),
|
||||
"timestamp": timestamp_ms,
|
||||
}),
|
||||
"timestamp": timestamp_ms * 1_000_000,
|
||||
}
|
||||
return {"payload": json.dumps(payload)}
|
||||
Reference in New Issue
Block a user