add_job_offer.py 32.9 KB
Newer Older
1
#!/usr/bin/env python3
2
# -*- coding: utf-8 -*-
3
4
5
6
7
"""
Job offer submission form.

WARNING: attachment has multiple copies in memory thus leading to a non
    negligible memory footprint.
Roland Denis's avatar
Roland Denis committed
8
9
10
11
12
13
14
15

FOR DEVELOPERS:

# General workflow (see process_form function)
1) The form data are put into a FormData instance
2) Form sanity is checked and a FormError is filled with potential errors
3) If there is an error, the form is displayed with additional error message
4) Otherwise, a job offer is filled using the JobOffer class
16
5) A new pull-request is created with the create_job_request function
Roland Denis's avatar
Roland Denis committed
17
18
19
20
21

# How to add a form field
1) if it can raises an error (ie an invalid form), first add a property to
    FormError class (contains the error message) and had a condition in the
    has_error method.
22
23
24
2) add a corresponding property to FormData class (with default value) and
    add (if necessary) a sanity check in the check method. Also check if
    the field is mandatory.
Roland Denis's avatar
Roland Denis committed
25
26
3) update the web form template and/or the update_html_form function.
    Don't forget to escape Jinja script so that the pelican pass don't
27
    remove it. Add `required` attribute and a `*` if necessary.
Roland Denis's avatar
Roland Denis committed
28
29
30
31
4) describe how to fill the FormData from the input request in the process_*
    functions.
5) add a corresponding property in JobOffer class
6) include this property in the id calculation (_calc_id method of JobOffer).
32
33
34
35
36
7) add a corresponding line in the job offer file template
    (content/job_offers/job_offer.md.template) and add corresponding line when
    launching the rendering (_render method of JobOffer).
8) update the Pelican job offer template (job_offer.html).
9) add corresponding line in the interface processors
37
   (process_cmdline and process_cgi so far).
Roland Denis's avatar
Roland Denis committed
38

39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# How has SimpleMDE editor been customized ?
1) Firstly, please consider using another editor since it seems to be
    abandoned.
2) OK, as you want. Install SimpleMDE using the Node.js package manager :
    npm install simplemde --save
    (you may need to install dependencies, I don't remember...)
3) Fix some bugs that prevent "compilation" of SimpleMDE:
    * in src/css/simplemde.css, line 121-122 (in the .editor-toolbar a block),
        add a space before `!important` for `text-decoration` and `color`
        properties.
    * in src/js/simplemde.js, remove the backslash before `-` in each regular
        expression, ie replace `\-` by `-`. Should be the case for lines 75,
        174, 904, 905 and 1012.
4) Disable HTML in Markdown syntax: in src/js/simplemde.js, add after
    `var marked = require("marked");` line 15, the following line :
    marked.setOptions({sanitize: true});
5) "Compile" the package by running `gulp` (install it using npm) from the root
    of SimpleMDE.
6) Enjoy the unreadable files in dist subfolder.

59
60
61
62
63
# What about security ?
1) access to *.template files should be denied in the web server configuration
    (e.g. through a .htaccess file, like the one already there).
2) in case of security issue (e.g. this script has been displayed on client side),
    consider revoking and updating the GITLAB_TOKEN below.
64
65
66
67
68
69

# Common errors:
- if script fails with error
    "AttributeError: module 'magic' has no attribute 'from_buffer'"
    you may want to install python-magic module ;) (otherwise, import magic will
    load Magic-file-extensions).
70
"""
71

Calcul Bot's avatar
Calcul Bot committed
72
73
import datetime
import os
Calcul Bot's avatar
Calcul Bot committed
74
import re
75
import sys
76
import io
77
78
79
import base64
import itertools
import pprint
80
81
import mimetypes
import magic
82
83
84
import hashlib
import pickle
import uuid
Calcul Bot's avatar
Calcul Bot committed
85

86
###############################################################################
Calcul Bot's avatar
Calcul Bot committed
87
# General configuration
88
89

from gitlab_config import * # For Gitlab, see gitlab_config.py
90

Calcul Bot's avatar
PEP8    
Calcul Bot committed
91
TEMPLATE_PATH = './'
92
93
94
TEMPLATE_JOB_OFFER_FORM = 'job_offer_form.html.template'
TEMPLATE_JOB_OFFER = 'job_offer.md.template'

Roland Denis's avatar
Roland Denis committed
95
96
97
98
99
100
101
102
JOBOFFER_TYPE = {
    'cdi': 'CDI',
    'cdd': 'CDD',
    'postdoc': 'Post-doctorat',
    'these': 'Thèse',
    'stage': 'Stage'
}

103
PELICAN_JOB_OFFER_PATH = 'content/job_offers'
104
ATTACHMENT_MIME_TYPE = ['application/pdf']
105
JOBOFFER_EXPIRATION_DELAY = datetime.timedelta(weeks=3*4)
Calcul Bot's avatar
PEP8    
Calcul Bot committed
106

107
108
109
110
111
112
113
114
115
116
117
118
# The folder where files associated to each submission are created.
# Should be a folder that depends on the current website to avoid collisions.
FLOOD_PATH = "./flood"

# Global flood limits (for all submissions)
FLOOD_GLOBAL_TIMEOUT = datetime.timedelta(minutes=10)
FLOOD_GLOBAL_LIMIT = 10

# Local flood limits (for one filler)
FLOOD_LOCAL_TIMEOUT = datetime.timedelta(minutes=5)
FLOOD_LOCAL_LIMIT = 3

119
120
121
122
123
124
125
126
127
###############################################################################
class Debug(object):
    """ Debugging parameters. """

    def __init__(self, verbose=False, offline=False, local=False):
        self.verbose = verbose  # Explains all options
        self.offline = offline  # Do not connect to Gitlab
        self.local = local      # Print job offer files locally

128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
###############################################################################
class FloodChecker():
    """ Anti-flood protection """

    class FloodStats():
        """ Local or global flood stats """
        def __init__(self):
            self.first = datetime.datetime.max
            self.last = datetime.datetime.min
            self.cnt = 0

        def add(self, dt):
            self.first = min(self.first, dt)
            self.last = max(self.last, dt)
            self.cnt += 1

        def __repr__(self):
            return "FloodStats(first={}, last={}, cnt={})".format(
                pprint.pformat(self.first),
                pprint.pformat(self.last),
                self.cnt
            )


    def __init__(self):
        self.dt_now = datetime.datetime.now() # To have a consistent current time

    def _read_files(self):
        """ Read submission files and clean outdated ones """

        self.global_stats = self.FloodStats()
        self.local_stats = dict()

        # Submission file name pattern: hexdigest followed by an uuid
        pattern = re.compile("(?P<datetime>[0-9-]+)_(?P<id>[a-z0-9]+)_[a-z0-9-]+")

        for entry in os.scandir(FLOOD_PATH):
            if entry.is_file():
                file_name_match = pattern.fullmatch(entry.name)
                if file_name_match:

                    try:
                        file_mtime = datetime.datetime.strptime(file_name_match.group('datetime'), "%Y-%m-%d-%H-%M-%S-%f")

                        # Clean if outdated file
                        if file_mtime < self.dt_now - max(FLOOD_GLOBAL_TIMEOUT, FLOOD_LOCAL_TIMEOUT):
                            try:
                                os.remove(entry.path)
                            except OSError:
                                pass # File may have been deleted by other script instance
                            continue

                        # Add to global stats
                        if file_mtime >= self.dt_now - FLOOD_GLOBAL_TIMEOUT:
                            self.global_stats.add(file_mtime)

                        # Add to local stats
                        if file_mtime >= self.dt_now - FLOOD_LOCAL_TIMEOUT:
                            self.local_stats.setdefault(file_name_match.group('id'), self.FloodStats()).add(file_mtime)

                    except OSError:
                        pass # File may have been deleted by other script instance

    def _client_id_hexdigest(self, client_id):
        """ Return the hexadecimal digest of the given client identification data """
        return hashlib.sha1(pickle.dumps(client_id)).hexdigest()

    def _submission_delay(self, stats, timeout, limit):
        """ Return delay before a form can be submitted (0 if no delay) """
        if stats.cnt <= limit:
            return datetime.timedelta()
        else:
            return max(datetime.timedelta(), timeout - (self.dt_now - stats.first))

    def _global_submission_delay(self):
        """ Return delay before a form can be submitted (0 if no delay) by anyone """
        return self._submission_delay(
            self.global_stats,
            FLOOD_GLOBAL_TIMEOUT,
            FLOOD_GLOBAL_LIMIT
        )

    def _local_submission_delay(self, client_id):
        """ Return delay before a form can be submitted (0 if no delay) by a given client """
        return self._submission_delay(
            self.local_stats.get(self._client_id_hexdigest(client_id), self.FloodStats()),
            FLOOD_LOCAL_TIMEOUT,
            FLOOD_LOCAL_LIMIT
        )

    def _create_submission_file(self, client_id):
        """ Add a file to register a valid submission """
        # Submission file name is composed of
        # - the current datetime
        # - hexadecimal digest of the client identification data (to ease dump to string and increase condifendiality)
        # - a random suffix to avoid generating same file name in multiple instances.
        self.file_name = "{}_{}_{}".format(
            datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S-%f"),
            self._client_id_hexdigest(client_id),
            uuid.uuid4()
        )

        # Touch the file
        with open(os.path.join(FLOOD_PATH, self.file_name), 'w'):
            pass

    def _remove_submission_file(self):
        """ Remove previously created submission file """
        os.remove(os.path.join(FLOOD_PATH, self.file_name))

    def approve_submission(self, client_id):
        """ Check if a new submission can be accepted. Return bool and waiting delay. """

        # First, create a submission file even if it will be deleted if submission is rejected
        # It avoids flooding at the exact same time.
        self._create_submission_file(client_id)
        self._read_files()

        # Calculate submission delays
        global_delay = self._global_submission_delay()
        local_delay = self._local_submission_delay(client_id)
        delay = max(global_delay, local_delay)

        if delay.total_seconds() == 0: # No delay => OK
            return True, delay

        else: # Otherwise => KO and submission file is removed
            flood_details = "[FLOOD] client_id={} client_hash={} global_stats={} global_delay={} local_stats={} local_delay={}".format(
                pprint.pformat(client_id),
                pprint.pformat(self._client_id_hexdigest(client_id)),
                pprint.pformat(self.global_stats),
                pprint.pformat(global_delay),
                pprint.pformat(self.local_stats),
                pprint.pformat(local_delay),
            )
            print(flood_details, file=sys.stderr)
            #TODO: mail

            self._remove_submission_file()

            return False, delay


271
272
273
###############################################################################
class FormData(object):
    """ Job offer data extracted from the form. """
Calcul Bot's avatar
Calcul Bot committed
274
275

    def __init__(self,
276
                 client_id=None, # Client identification data
277
278
279
                 title='',
                 description='',
                 author='',
280
                 employer='',
281
                 email='',
Roland Denis's avatar
Roland Denis committed
282
                 job_type='',
283
284
285
                 location='',
                 duration='',
                 website='',
286
                 expiration='',
287
                 attachment_content=None,
288
                 attachment_name=''):
289

290
        self.client_id = client_id
291
292
293
        self.title = title.strip()
        self.description = description.strip()
        self.author = author.strip()
294
        self.employer = employer.strip()
295
        self.email = email.strip()
Roland Denis's avatar
Roland Denis committed
296
        self.job_type = job_type
297
298
299
        self.location = location.strip()
        self.duration = duration.strip()
        self.website = website.strip()
300
        self.expiration = expiration if expiration else (datetime.datetime.now() + JOBOFFER_EXPIRATION_DELAY).strftime('%Y-%m-%d')
301
302
303
304
305
306
        self.attachment_content = attachment_content
        self.attachment_name = attachment_name

    def has_attachment(self):
        """ True if there is a given attachment. """
        return self.attachment_content is not None
307

308
309
310
311
312
313
314
315
316
317
318
319
    def check(self):
        """ Checks form validity and returns errors.

        If multi-language is needed, we could replace the
        error messages by codes.
        """

        errors = FormError()

        # Title must be set
        if not self.title:
            errors.title = 'Titre manquant'
Calcul Bot's avatar
Calcul Bot committed
320

321
322
323
324
        # Author must be set
        if not self.author:
            errors.author = 'Nom manquant'

325
326
327
328
        # Employer must be set
        if not self.employer:
            errors.employer = 'Employeur manquant'

329
330
331
332
333
        # Description must be set
        if not self.description:
            errors.description = 'Description manquante'

        # Checking email
334
335
336
337
338
        if self.email:
            if not re.match(r'^[^@]+@[^@]+\.[^@]+$', self.email):
                errors.email = 'Adresse mail invalide'
        else:
            errors.email = 'Adresse mail manquante'
339

Roland Denis's avatar
Roland Denis committed
340
341
342
343
        # Checking job type
        if self.job_type not in JOBOFFER_TYPE:
            errors.job_type = "Type d'offre d'emploi invalide"

344
        # Checking attachment type
345
346
347
348
349
350
351
        if self.has_attachment():
            # Getting mime type from magic numbers.
            attachment_mime = magic.from_buffer(self.attachment_content, mime=True)

            # Checking allowed mime types.
            if attachment_mime not in ATTACHMENT_MIME_TYPE:
                errors.attachment = 'Type de fichier invalide'
352

353
354
355
356
357
358
359
360
        # Checking expiration date
        try:
            expiration_date = datetime.datetime.strptime(self.expiration, '%Y-%m-%d')
            if expiration_date <= datetime.datetime.now():
                errors.expiration = "L'offre doit expirer dans le futur"
        except:
            errors.expiration = "Format de date invalide"

361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
        return errors

    def __str__(self):
        """ String representation with attachment length instead of his content. """
        return pprint.pformat(dict(itertools.chain(
            {(k, v) for k, v in vars(self).items() if k != 'attachment_content'},
            {('attachment_content_length', len(self.attachment_content) if self.has_attachment() else None)}
        )))


###############################################################################
class FormError(object):
    """ Error messages in the form. """

    def __init__(self,
Roland Denis's avatar
Roland Denis committed
376
377
378
                 title='',
                 description='',
                 author='',
379
                 employer='',
Roland Denis's avatar
Roland Denis committed
380
                 email='',
Roland Denis's avatar
Roland Denis committed
381
                 job_type='',
382
                 expiration='',
Roland Denis's avatar
Roland Denis committed
383
                 attachment=''):
384

Calcul Bot's avatar
Calcul Bot committed
385
386
        self.title = title
        self.description = description
387
        self.author = author
388
        self.employer = employer
389
        self.email = email
Roland Denis's avatar
Roland Denis committed
390
        self.job_type = job_type
391
        self.expiration = expiration
Calcul Bot's avatar
Calcul Bot committed
392
        self.attachment = attachment
Calcul Bot's avatar
Calcul Bot committed
393

394
395
396
    def has_error(self):
        """ True if there is any error set. """
        return (
Roland Denis's avatar
Roland Denis committed
397
398
399
            self.title
            or self.description
            or self.author
400
            or self.employer
Roland Denis's avatar
Roland Denis committed
401
            or self.email
Roland Denis's avatar
Roland Denis committed
402
            or self.job_type
403
            or self.expiration
Roland Denis's avatar
Roland Denis committed
404
            or self.attachment
405
406
407
408
409
410
411
412
        )

    @property
    def general(self):
        """ Error displayed at the top of the form. """
        if self.has_error():
            return 'Un ou plusieurs champs sont mal renseignés.'
        else:
Roland Denis's avatar
Roland Denis committed
413
            return ''
414
415
416
417
418
419
420

    def __str__(self):
        """ String representation. """
        return pprint.pformat(dict(itertools.chain(
            {('general', self.general)},
            vars(self).items()
        )))
421

422
423
424
425
426
427
428
429
430

###############################################################################
class GitlabFile(object):
    """ File commited to Gitlab repo. """

    def __init__(self, file_path, encoding, content):
        self.file_path = file_path
        self.encoding = encoding
        self.content = content
431

432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
    def __str__(self):
        """ String representation. """
        return pprint.pformat(vars(self))


###############################################################################
class JobOffer(object):
    """ Job offer formated for a Pelican website. """

    def __init__(self, form_data):

        # Copying form fields
        self.title = form_data.title
        self.description = form_data.description
        self.author = form_data.author
447
        self.employer = form_data.employer
448
        self.email = form_data.email
Roland Denis's avatar
Roland Denis committed
449
        self.job_type = form_data.job_type
450
        self.attachment_user_name = form_data.attachment_name
451
452
453
        self.location = form_data.location
        self.duration = form_data.duration
        self.website = form_data.website
454
        self.expiration = datetime.datetime.strptime(form_data.expiration, '%Y-%m-%d')
455
456

        # Attachment special care
457
        if form_data.has_attachment():
458
459
460
461
462
            # Ensuring valid extension
            attachment_mime = magic.from_buffer(form_data.attachment_content, mime=True)
            self.attachment_ext = mimetypes.guess_extension(attachment_mime)

            # Encoding attachment in base64
463
464
            self.attachment_content = base64.b64encode(
                form_data.attachment_content).decode()
465

466
467
468
469
470
471
472
473
474
475
476
        else:
            self.attachment_content = None

        # Submission date
        self.date = datetime.datetime.now()

        # Generate unique id
        self.id = self._calc_id()

        # Rendering job offer description
        self.main_content = self._render()
Calcul Bot's avatar
Calcul Bot committed
477

Calcul Bot's avatar
Calcul Bot committed
478
479

    def has_attachment(self):
480
        """ True if the job-offer has an attachment. """
481
        return self.attachment_content is not None
482
483
484
485

    @property
    def name(self):
        """ Base name of the job offer. """
486
        return 'job_{self.id}'.format(self=self)
487
488
489
490

    @property
    def file_name(self):
        """ Name of the job-offer description file. """
491
        return '{self.name}.md'.format(self=self)
492
493
494
495
496

    @property
    def attachment_name(self):
        """ Name of the job-offer attachment. """
        if not self.has_attachment():
497
            return ''
498

499
        return '{self.name}_attachment{self.attachment_ext}'.format(self=self)
500
501
502
503
504
505
506
507
508

    @property
    def blog_file(self):
        """ The main file of the blog entry. """
        return GitlabFile(
            file_path=PELICAN_JOB_OFFER_PATH + '/' + self.file_name,
            encoding='text',
            content=self.main_content
        )
Calcul Bot's avatar
Calcul Bot committed
509

510
511
512
    @property
    def blog_attachment(self):
        """ Return the job offer attachment. """
513
        if not self.has_attachment():
514
            return None
Calcul Bot's avatar
Calcul Bot committed
515

516
517
518
        return GitlabFile(
            file_path=PELICAN_JOB_OFFER_PATH + '/' + self.attachment_name,
            encoding='base64',
519
            content=self.attachment_content
520
        )
Calcul Bot's avatar
Calcul Bot committed
521

522
523
524
525
526
527
528
    @property
    def gitlab_files(self):
        """ The files to be commited in Gitlab repo. """
        yield self.blog_file

        if self.has_attachment():
            yield self.blog_attachment
529

530
531
532
533
    def _calc_id(self):
        """ Calculate job offer id. """
        # MD5 hash
        import hashlib
534
        m = hashlib.md5()
535
536
537

        # Feeding the hash algo with the job_offer fields
        m.update(b'|'.join(str(getattr(self, field)).encode() for field in
538
            ('title', 'job_type', 'author', 'employer', 'date', 'description', 'email', 'attachment_user_name', 'location', 'duration', 'website', 'expiration')
539
        ))
540

541
542
        # Feeding the hash algo with the job_offer attachment
        if self.has_attachment():
543
            m.update(b'|' + self.attachment_content.encode())
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566

        return m.hexdigest()

    def _render(self):
        """ Render a job offer to a Markdown file using Jinja2. """

        from jinja2 import Environment, FileSystemLoader

        # Jinja2 environment
        env = Environment(
            loader=FileSystemLoader(TEMPLATE_PATH),  # Path to the templates
            autoescape=False                         # Manual HTML escaping
        )

        # Adding custom filters
        env.filters.update({
            'markdown': filter_markdown,
            'pelican': filter_pelican,
            'linebreaks': filter_linebreaks,
            'datetime': filter_datetime
        })

        # Gets job offer template
567
        template = env.get_template(TEMPLATE_JOB_OFFER)
Calcul Bot's avatar
Tests    
Calcul Bot committed
568

569
570
571
572
573
574
        # Rendering
        return template.render(
            title=self.title,
            date=self.date,
            slug=self.name,
            attachment_name=self.attachment_name,
Roland Denis's avatar
Roland Denis committed
575
576
            description=self.description,
            job_type=JOBOFFER_TYPE[self.job_type],
577
            tag=self.job_type,
578
            author=self.author,
579
            employer=self.employer,
580
581
582
583
            email=self.email,
            location=self.location,
            duration=self.duration,
            website=self.website,
584
            expiration=self.expiration,
585
586
587
588
589
        )


###############################################################################
# Jinja2 custom filters
Calcul Bot's avatar
Tests    
Calcul Bot committed
590

Calcul Bot's avatar
Calcul Bot committed
591
592
593
594
def filter_markdown(text):
    """ Escapes all special characters of Markdown syntax. """
    return re.sub('([' + re.escape(r'\`*_{}[]()#>+-.!|') + '])', r'\\\1', text)

Calcul Bot's avatar
PEP8    
Calcul Bot committed
595

Calcul Bot's avatar
Calcul Bot committed
596
597
598
599
def filter_pelican(text):
    """
    Escapes special characters of Pelican while keeping most of
    the Markdown syntax.
600
601
602
603
604
605
606
607

    {} to avoid {filename}, {attach} and such special tags
    |  since it is the old syntax for {}
    >  (blockquote) since it will be escaped for html
    !  for inline image links.

    TODO: also escape named links ?
    TODO: modify SimpleMDE configuration accordingly
Calcul Bot's avatar
Calcul Bot committed
608
    """
609
    return re.sub('([' + re.escape(r'{}>|!') + '])', r'\\\1', text)
Calcul Bot's avatar
Calcul Bot committed
610

Calcul Bot's avatar
PEP8    
Calcul Bot committed
611

Calcul Bot's avatar
Calcul Bot committed
612
613
614
615
def filter_linebreaks(text, replacement='<br>'):
    """ Replaces newlines and carriage returns by the given string. """
    return re.sub('[\r\n]+', replacement, text)

Calcul Bot's avatar
PEP8    
Calcul Bot committed
616
617

def filter_datetime(date, date_format='%Y-%m-%d %H:%M'):
618
    """ Format a date and time. """
Calcul Bot's avatar
PEP8    
Calcul Bot committed
619
620
    return date.strftime(date_format)

Calcul Bot's avatar
Calcul Bot committed
621

622
###############################################################################
623
def create_job_request(job_offer, debug):
Calcul Bot's avatar
Calcul Bot committed
624
625
626
627
    """ Creates a merge request for the given job offer.

    TODO: generates job id in this function.
    """
Calcul Bot's avatar
PEP8    
Calcul Bot committed
628

629
    # Connecting to Gitlab
gouarin's avatar
gouarin committed
630
631
    gitlab_private_token = GITLAB_TOKEN
    if gitlab_private_token is None:
632
633
        print("[ERROR] GITLAB_PRIVATE_TOKEN environment variable not set!", file=sys.stderr)
        raise
Matthieu Boileau's avatar
Matthieu Boileau committed
634

635
    if debug.verbose:
636
        print('[DEBUG] Gitlab connection to {} '.format(GITLAB_URL) +
Matthieu Boileau's avatar
Matthieu Boileau committed
637
              'with token {}'.format(gitlab_private_token), file=sys.stderr)
638
639

    if not debug.offline:
640
        import gitlab
Matthieu Boileau's avatar
Matthieu Boileau committed
641

Calcul Bot's avatar
PEP8    
Calcul Bot committed
642
643
        gl = gitlab.Gitlab(
            GITLAB_URL,
Matthieu Boileau's avatar
Matthieu Boileau committed
644
            private_token=gitlab_private_token,
Calcul Bot's avatar
PEP8    
Calcul Bot committed
645
646
            api_version=4
        )
647
648

    # Checking connexion ?
Calcul Bot's avatar
Calcul Bot committed
649

650
651
    # Branch name associated to this job offer
    branch_name = job_offer.name
Calcul Bot's avatar
Calcul Bot committed
652
653

    # Accessing project
654
    if debug.verbose:
655
        print('[DEBUG] Accessing bot projet {}'.format(GITLAB_SOURCE_ID), file=sys.stderr)
656
657

    if not debug.offline:
Calcul Bot's avatar
PEP8    
Calcul Bot committed
658
        project = gl.projects.get(GITLAB_SOURCE_ID)
Calcul Bot's avatar
Calcul Bot committed
659

Calcul Bot's avatar
Calcul Bot committed
660
    # Creating new branch
661
    if debug.verbose:
662
        print('[DEBUG] Creating branch {} from master'.format(branch_name), file=sys.stderr)
663
664

    if not debug.offline:
Calcul Bot's avatar
PEP8    
Calcul Bot committed
665
666
        branch = project.branches.create({
            "branch": branch_name,
667
            "ref": GITLAB_REF_BRANCH
Calcul Bot's avatar
PEP8    
Calcul Bot committed
668
        })
669

Calcul Bot's avatar
Calcul Bot committed
670
671
    # Creating commit
    data = {
Calcul Bot's avatar
PEP8    
Calcul Bot committed
672
673
        "branch": branch_name,
        "commit_message": "Adding new job offer {}".format(branch_name),
674
        "actions": []
Calcul Bot's avatar
Calcul Bot committed
675
676
    }

677
678
    # Adding files to the commit
    for gitlab_file in job_offer.gitlab_files:
Calcul Bot's avatar
Calcul Bot committed
679
680
        data['actions'].append({
            'action': 'create',
681
682
683
            'file_path': gitlab_file.file_path,
            'encoding': gitlab_file.encoding,
            'content': gitlab_file.content
Calcul Bot's avatar
Calcul Bot committed
684
685
686
        })

    # Committing
687
    if debug.verbose:
Roland Denis's avatar
Roland Denis committed
688
        print('[DEBUG] Committing:', file=sys.stderr)
689
690
691
692
        print(pprint.pformat(data), file=sys.stderr)

    if not debug.offline:
        commit = project.commits.create(data)
Calcul Bot's avatar
Calcul Bot committed
693

Calcul Bot's avatar
Calcul Bot committed
694
    # Creating merge request
695
    pr_data = {
Calcul Bot's avatar
Calcul Bot committed
696
        'source_branch': branch_name,
Calcul Bot's avatar
PEP8    
Calcul Bot committed
697
698
        'target_branch': GITLAB_TARGET_BRANCH,
        'target_project_id': GITLAB_TARGET_ID,
Calcul Bot's avatar
Calcul Bot committed
699
        'title': 'New job offer {}'.format(branch_name),
700
        'description': job_offer.main_content.replace("\n", "  \n"),
701
702
        'remove_source_branch': True,
        'labels': GITLAB_LABELS
703
704
    }

705
    if debug.verbose:
Roland Denis's avatar
Roland Denis committed
706
        print('[DEBUG] Creating PR:', file=sys.stderr)
707
708
709
        print(pprint.pformat(pr_data), file=sys.stderr)

    if not debug.offline:
710
        project.mergerequests.create(pr_data)
711
712
713
714
715
716
717
718
719
720
721
722
723
724


###############################################################################
def local_write_job(job_offer):
    """ Write job offer locally. """

    for gitlab_file in job_offer.gitlab_files:
        if gitlab_file.encoding == 'text':
            with open(os.path.basename(gitlab_file.file_path), 'w') as f:
                f.write(gitlab_file.content)
        else:
            with open(os.path.basename(gitlab_file.file_path), 'wb') as f:
                f.write(base64.b64decode(gitlab_file.content.encode()))

725

726
###############################################################################
727
def update_html_form(form_data=FormData(), errors=FormError(), debug=Debug(), internal_error=False, success=False, flood_error=False, flood_delay=None):
728
729
    """ Update the form with error messages. """

Roland Denis's avatar
Roland Denis committed
730
    from jinja2 import Environment, FileSystemLoader
Roland Denis's avatar
Roland Denis committed
731

Roland Denis's avatar
Roland Denis committed
732
733
734
735
736
    # Jinja2 environment
    env = Environment(
        loader=FileSystemLoader(TEMPLATE_PATH),  # Path to the templates
        autoescape=True                          # Auto HTML escaping
    )
737

Roland Denis's avatar
Roland Denis committed
738
739
    # Gets job offer template
    template = env.get_template(TEMPLATE_JOB_OFFER_FORM)
740

Roland Denis's avatar
Roland Denis committed
741
    # Allowed attachment types
Roland Denis's avatar
Roland Denis committed
742
    file_accept = (
743
744
745
746
747
748
749
        ','.join([ext
            for mime_type in ATTACHMENT_MIME_TYPE
            for ext in mimetypes.guess_all_extensions(mime_type)])
        + ','
        + ','.join(ATTACHMENT_MIME_TYPE)
    )

Roland Denis's avatar
Roland Denis committed
750
751
752
753
    # Rendering
    print(template.render(
        form=form_data,
        errors=errors,
Roland Denis's avatar
Roland Denis committed
754
        file_accept=file_accept,
755
756
757
        job_type_list=JOBOFFER_TYPE,
        internal_error=internal_error,
        success=success,
758
759
        flood_error=flood_error,
        flood_delay=flood_delay,
Roland Denis's avatar
Roland Denis committed
760
    ))
761

762
763


764
765
766
###############################################################################
def process_form(form_data, debug=Debug()):
    """ Check and submit job-offer. """
767

768
    errors = FormError()
769

770
771
772
773
    try:
        # Displaying form data in debug mode
        if debug.verbose:
            print('[DEBUG] Form data: {}\n'.format(form_data), file=sys.stderr)
774

775
776
777
778
779
        # Checking form validity
        errors = form_data.check()

        if debug.verbose:
            print('[DEBUG] Form errors: {}\n'.format(errors), file=sys.stderr)
780

781
782
783
784
        if errors.has_error():
            update_html_form(form_data, errors, debug)
            return

785
786
787
788
789
790
        # Checking undergoing flood
        is_flood_ok, flood_delay = FloodChecker().approve_submission(form_data.client_id)
        if not is_flood_ok:
            update_html_form(form_data, errors, debug, flood_error=True, flood_delay=flood_delay)
            return

791
792
793
794
795
796
797
798
        # Creating job offer
        job_offer = JobOffer(form_data)

        # Creating pull request
        if debug.local:
            local_write_job(job_offer)
        else:
            create_job_request(job_offer, debug)
799

800
801
        # Success page
        update_html_form(debug=debug, success=True)
802

803
804
805
806
807
808
    except Exception as error:
        # Internal error
        import traceback
        print("[ERROR] Error while submitting job offer: {}".format(error), file=sys.stderr)
        print("[ERROR] {}".format(traceback.format_exc()), file=sys.stderr)
        update_html_form(form_data, errors, debug, internal_error=True)
Calcul Bot's avatar
Calcul Bot committed
809
810


Roland Denis's avatar
Roland Denis committed
811

812
813
814
815
###############################################################################
def process_cmdline():
    """ Process command-line arguments. """

Calcul Bot's avatar
Calcul Bot committed
816
817
818
819
    import argparse

    parser = argparse.ArgumentParser()

820
    # Debug options
821
    debug_parser = parser.add_argument_group('debug arguments')
Calcul Bot's avatar
PEP8    
Calcul Bot committed
822
    debug_parser.add_argument(
823
824
825
        '-v', '--verbose',
        action='store_true',
        help='output process informations'
Calcul Bot's avatar
PEP8    
Calcul Bot committed
826
827
    )
    debug_parser.add_argument(
828
        '-o', '--offline',
Calcul Bot's avatar
PEP8    
Calcul Bot committed
829
830
831
        action='store_true',
        help='Disable Gitlab connection and pull-request creation'
    )
832
833
834
835
836
    debug_parser.add_argument(
        '-l', '--local',
        action='store_true',
        help='Print job offer files locally (implies offline)'
    )
837
838

    # Form options
839
    form_parser = parser.add_argument_group('form arguments')
Roland Denis's avatar
Roland Denis committed
840
841
842
843
844
    form_parser.add_argument(
        '--type',
        default='',
        help='Offer type within ' + ','.join(JOBOFFER_TYPE.keys())
    )
Calcul Bot's avatar
PEP8    
Calcul Bot committed
845
846
    form_parser.add_argument(
        '--title',
847
        default='',
Calcul Bot's avatar
PEP8    
Calcul Bot committed
848
849
850
        help='Offer title'
    )
    form_parser.add_argument(
851
852
853
        '--author',
        default='',
        help='Offer author name'
Calcul Bot's avatar
PEP8    
Calcul Bot committed
854
    )
855
856
857
858
859
    form_parser.add_argument(
        '--employer',
        default='',
        help='Employer'
    )
Roland Denis's avatar
Roland Denis committed
860
861
862
863
864
    form_parser.add_argument(
        '--email',
        default='',
        help='Offer author email'
    )
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
    form_parser.add_argument(
        '--location',
        default='',
        help='Job working location'
    )
    form_parser.add_argument(
        '--duration',
        default='',
        help='Job duration'
    )
    form_parser.add_argument(
        '--website',
        default='',
        help='Web page'
    )
880
881
882
883
884
    form_parser.add_argument(
        '--expiration',
        default='',
        help='Offer expiration date'
    )
Calcul Bot's avatar
PEP8    
Calcul Bot committed
885
    form_parser.add_argument(
886
887
888
        '--description',
        default='',
        help='Offer description'
Calcul Bot's avatar
PEP8    
Calcul Bot committed
889
    )
890
891
    form_parser.add_argument(
        '--attachment',
892
        default='',
893
894
        help='File attachment'
    )
895
896
897
898
899
    form_parser.add_argument(
        '--client_id',
        default=None,
        help='Client id'
    )
900
901
902

    # Parsing arguments
    args = parser.parse_args()
Calcul Bot's avatar
Calcul Bot committed
903

904
905
906
    # Set offline to true if local is true
    args.offline = args.offline or args.local

907
908
909
910
    # Debug options
    debug = Debug(verbose=args.verbose, offline=args.offline, local=args.local)

    # Reading attachment
911
    if args.attachment:
912
913
        with open(args.attachment, 'rb') as f:
            attachment_content = f.read()
914
        attachment_name = os.path.basename(args.attachment)
915
916
    else:
        attachment_content = None
917
        attachment_name = ''
918

919
    # Creating form data
920
    form_data = FormData(
Roland Denis's avatar
Roland Denis committed
921
        job_type=args.type,
922
        title=args.title,
923
        author=args.author,
924
        employer=args.employer,
Roland Denis's avatar
Roland Denis committed
925
        email=args.email,
926
927
928
        location=args.location,
        duration=args.duration,
        website=args.website,
929
        expiration=args.expiration,
930
        description=args.description,
931
        attachment_name=attachment_name,
932
933
        attachment_content=attachment_content,
        client_id=args.client_id,
934
935
    )

936
937
    # Continue submission process
    process_form(form_data, debug)
Calcul Bot's avatar
Calcul Bot committed
938

Calcul Bot's avatar
PEP8    
Calcul Bot committed
939

940
941
942
943
###############################################################################
def process_cgi():
    """ Process cgi arguments. """

944
945
946
947
948
949
    import cgi

    # Accessing CGI form data
    cgi_form = cgi.FieldStorage()

    # Default form data
950
    form_data = FormData()
951
952
953

    # Filling form fields
    if 'title' in cgi_form:
954
        form_data.title = cgi_form['title'].value
955

956
    if 'author' in cgi_form:
Roland Denis's avatar
Roland Denis committed
957
        form_data.author = cgi_form['author'].value
958

959
960
961
    if 'employer' in cgi_form:
        form_data.employer = cgi_form['employer'].value

962
    if 'email' in cgi_form:
Roland Denis's avatar
Roland Denis committed
963
        form_data.email = cgi_form['email'].value
964
965

    if 'description' in cgi_form:
Roland Denis's avatar
Roland Denis committed
966
        form_data.description = cgi_form['description'].value
967

Roland Denis's avatar
Roland Denis committed
968
    if 'job_type' in cgi_form:
Roland Denis's avatar
Roland Denis committed
969
        form_data.job_type = cgi_form['job_type'].value
970

971
972
973
974
975
    if 'location' in cgi_form:
        form_data.location = cgi_form['location'].value

    if 'duration' in cgi_form:
        form_data.duration = cgi_form['duration'].value
976

977
978
    if 'website' in cgi_form:
        form_data.website = cgi_form['website'].value
Roland Denis's avatar
Roland Denis committed
979

980
981
982
    if 'expiration' in cgi_form:
        form_data.expiration = cgi_form['expiration'].value

983
984
    # Checking attachment
    # TODO: checking 'done' attribute for transfer error
985
    if 'file' in cgi_form:
986
        file_item = cgi_form["file"]
987
988
        if file_item.filename:
            form_data.attachment_name = file_item.filename
Roland Denis's avatar
Roland Denis committed
989
            form_data.attachment_content = file_item.file.read(-1)
990

991
992
993
    # Client identification data (IP)
    form_data.client_id = os.getenv("REMOTE_ADDR")

994
995
996
997
998
999
1000
    # Checking if the form was submitted
    if 'submit' in cgi_form:
        # Continue submission process
        process_form(form_data)
    else:
        # Displaying form without errors, possibly pre-filled through
        #   "GET" parameters (eg http://.../add_job_offer?job_type=cdd)
For faster browsing, not all history is shown. View entire blame